[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1027920347 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003350] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008691] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.011163] LSM: Security Framework initializing [ 0.012075] Yama: becoming mindful. [ 0.013052] SELinux: Initializing. [ 0.014082] *** VALIDATE selinux *** [ 0.023755] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029085] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034010] *** VALIDATE tmpfs *** [ 0.035460] *** VALIDATE proc *** [ 0.037280] *** VALIDATE cgroup *** [ 0.038011] *** VALIDATE cgroup2 *** [ 0.040376] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043035] Spectre V2 : User space: Vulnerable [ 0.044009] Speculative Store Bypass: Vulnerable [ 0.048554] debug: unmapping init [mem 0xffffffff99859000-0xffffffff99860fff] [ 0.051550] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053019] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054034] ... version: 2 [ 0.055023] ... bit width: 48 [ 0.056013] ... generic registers: 4 [ 0.057012] ... value mask: 0000ffffffffffff [ 0.058015] ... max period: 00007fffffffffff [ 0.059017] ... fixed-purpose events: 3 [ 0.060015] ... event mask: 000000070000000f [ 0.061548] rcu: Hierarchical SRCU implementation. [ 0.064350] smp: Bringing up secondary CPUs ... [ 0.065997] x86: Booting SMP configuration: [ 0.066042] .... node #0, CPUs: #1 #2 #3 [ 0.072281] smp: Brought up 1 node, 4 CPUs [ 0.074013] smpboot: Max logical packages: 1 [ 0.075024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233087] node 0 deferred pages initialised in 156ms [ 0.238178] devtmpfs: initialized [ 0.239496] x86/mm: Memory block size: 128MB [ 0.243835] gcov: version magic: 0x41383552 [ 0.246795] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252648] pinctrl core: initialized pinctrl subsystem [ 0.255165] [ 0.255832] ************************************************************* [ 0.258009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261011] ** ** [ 0.264014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266010] ** ** [ 0.268105] ** This means that this kernel is built to expose internal ** [ 0.271013] ** IOMMU data structures, which may compromise security on ** [ 0.274014] ** your system. ** [ 0.280021] ** ** [ 0.281016] ** If you see this message and you are not debugging the ** [ 0.285015] ** kernel, report this immediately to your vendor! ** [ 0.288011] ** ** [ 0.292015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.295017] ************************************************************* [ 0.299949] NET: Registered protocol family 16 [ 0.302670] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.303000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305145] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.313235] cpuidle: using governor menu [ 0.315000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.318816] PCI: Using configuration type 1 for base access [ 0.322424] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.334188] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.337061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340102] cryptd: max_cpu_qlen set to 1000 [ 0.345173] ACPI: Added _OSI(Module Device) [ 0.346021] ACPI: Added _OSI(Processor Device) [ 0.347037] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.348000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.359088] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.368445] ACPI: Interpreter enabled [ 0.369055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.371010] ACPI: Using IOAPIC for interrupt routing [ 0.375112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.377128] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.387365] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.390059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.392022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.397092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.433306] acpiphp: Slot [2] registered [ 0.436289] acpiphp: Slot [5] registered [ 0.443435] acpiphp: Slot [6] registered [ 0.447448] acpiphp: Slot [7] registered [ 0.454220] acpiphp: Slot [8] registered [ 0.465024] acpiphp: Slot [9] registered [ 0.470149] acpiphp: Slot [10] registered [ 0.472082] acpiphp: Slot [3] registered [ 0.474133] acpiphp: Slot [4] registered [ 0.503157] acpiphp: Slot [11] registered [ 0.504000] acpiphp: Slot [12] registered [ 0.504000] acpiphp: Slot [13] registered [ 0.504000] acpiphp: Slot [14] registered [ 0.504000] acpiphp: Slot [15] registered [ 0.504000] acpiphp: Slot [16] registered [ 0.505165] acpiphp: Slot [17] registered [ 0.508153] acpiphp: Slot [18] registered [ 0.509162] acpiphp: Slot [19] registered [ 0.511082] acpiphp: Slot [20] registered [ 0.512179] acpiphp: Slot [21] registered [ 0.513081] acpiphp: Slot [22] registered [ 0.515107] acpiphp: Slot [23] registered [ 0.516183] acpiphp: Slot [24] registered [ 0.517070] acpiphp: Slot [25] registered [ 0.519077] acpiphp: Slot [26] registered [ 0.520084] acpiphp: Slot [27] registered [ 0.521066] acpiphp: Slot [28] registered [ 0.523102] acpiphp: Slot [29] registered [ 0.524092] acpiphp: Slot [30] registered [ 0.525081] acpiphp: Slot [31] registered [ 0.527079] PCI host bridge to bus 0000:00 [ 0.528015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.530022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.532020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.535031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.537024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.539044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.540165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.543074] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.546264] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.555710] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.560602] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.563024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.565016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.567021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.570750] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.572799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.575035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.577947] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.582881] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.593017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.595865] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.600000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.729022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.745019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.795037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.803401] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.823022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.861023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.886022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.905189] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.916021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.927024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.950023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.965122] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.978033] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.984018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.998022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.008255] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.015018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.019019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.020000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.020000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.024127] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.028018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.043023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.054000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.057000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.059472] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.061507] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.064228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.069176] iommu: Default domain type: Passthrough [ 1.071432] SCSI subsystem initialized [ 1.072222] ACPI: bus type USB registered [ 1.073130] usbcore: registered new interface driver usbfs [ 1.076096] usbcore: registered new interface driver hub [ 1.077094] usbcore: registered new device driver usb [ 1.078196] pps_core: LinuxPPS API ver. 1 registered [ 1.080015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.083129] PTP clock support registered [ 1.085569] EDAC MC: Ver: 3.0.0 [ 1.086000] PCI: Using ACPI for IRQ routing [ 1.086901] NetLabel: Initializing [ 1.087011] NetLabel: domain hash size = 128 [ 1.089016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.092299] NetLabel: unlabeled traffic allowed by default [ 1.095084] vgaarb: loaded [ 1.097046] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.099014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.112143] clocksource: Switched to clocksource kvm-clock [ 1.257390] VFS: Disk quotas dquot_6.6.0 [ 1.260736] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.266041] *** VALIDATE ramfs *** [ 1.268505] *** VALIDATE hugetlbfs *** [ 1.272744] pnp: PnP ACPI init [ 1.277143] pnp: PnP ACPI: found 6 devices [ 1.308472] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.316143] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.321457] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.326078] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.346134] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.349989] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.354054] NET: Registered protocol family 2 [ 1.358779] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.371397] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.379316] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.389605] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.395977] TCP: Hash tables configured (established 65536 bind 65536) [ 1.400146] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.405382] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.412136] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.418960] NET: Registered protocol family 1 [ 1.424694] RPC: Registered named UNIX socket transport module. [ 1.426963] RPC: Registered udp transport module. [ 1.428518] RPC: Registered tcp transport module. [ 1.430298] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.439727] NET: Registered protocol family 44 [ 1.443526] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.446432] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.448826] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.451830] PCI: CLS 0 bytes, default 64 [ 1.454167] Unpacking initramfs... [ 3.763949] debug: unmapping init [mem 0xffff97e2fcc54000-0xffff97e2fffbffff] [ 3.770115] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.772185] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.776231] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.323716] Initialise system trusted keyrings [ 4.325503] Key type blacklist registered [ 4.327428] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.336527] zbud: loaded [ 4.339526] *** VALIDATE nfs *** [ 4.341179] *** VALIDATE nfs4 *** [ 4.342915] pstore: using deflate compression [ 4.347443] Platform Keyring initialized [ 4.479625] NET: Registered protocol family 38 [ 4.482391] Key type asymmetric registered [ 4.484223] Asymmetric key parser 'x509' registered [ 4.486635] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.490531] io scheduler mq-deadline registered [ 4.492723] io scheduler kyber registered [ 4.494977] io scheduler bfq registered [ 4.497689] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.501690] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.505447] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.509175] ACPI: Power Button [PWRF] [ 4.519505] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.534332] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.560433] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.572623] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.613361] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.668471] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.697607] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.701982] Non-volatile memory driver v1.3 [ 4.703387] Linux agpgart interface v0.103 [ 4.731163] virtio_blk virtio1: [vda] 134728 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.735659] vda: detected capacity change from 0 to 68980736 [ 4.757925] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.761294] vdb: detected capacity change from 0 to 1073741824 [ 4.775231] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.778237] vdc: detected capacity change from 0 to 2621440000 [ 4.795765] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.798480] vdd: detected capacity change from 0 to 2621440000 [ 4.814706] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.817526] vde: detected capacity change from 0 to 4294967296 [ 4.852457] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.854899] vdf: detected capacity change from 0 to 4294967296 [ 4.861496] libphy: Fixed MDIO Bus: probed [ 4.869970] usbcore: registered new interface driver usbserial_generic [ 4.872052] usbserial: USB Serial support registered for generic [ 4.874040] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.877906] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.879438] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.881920] mousedev: PS/2 mouse device common for all mice [ 4.885223] rtc_cmos 00:05: RTC can wake from S4 [ 4.888548] rtc_cmos 00:05: registered as rtc0 [ 4.890055] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.893801] intel_pstate: CPU model not supported [ 4.897899] hid: raw HID events driver (C) Jiri Kosina [ 4.899639] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.900754] usbcore: registered new interface driver usbhid [ 4.909084] usbhid: USB HID core driver [ 4.910564] drop_monitor: Initializing network drop monitor service [ 4.912698] Initializing XFRM netlink socket [ 4.915077] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.915985] NET: Registered protocol family 10 [ 4.921938] Segment Routing with IPv6 [ 4.923106] NET: Registered protocol family 17 [ 4.925202] mpls_gso: MPLS GSO support [ 4.932078] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.935792] RAS: Correctable Errors collector initialized. [ 4.935826] AVX version of gcm_enc/dec engaged. [ 4.935827] AES CTR mode by8 optimization enabled [ 5.016667] sched_clock: Marking stable (5016451132, 0)->(6723240053, -1706788921) [ 5.020185] registered taskstats version 1 [ 5.021840] Loading compiled-in X.509 certificates [ 5.023767] zswap: loaded using pool lzo/zbud [ 5.046623] Key type big_key registered [ 5.057896] Key type encrypted registered [ 5.059291] ima: No TPM chip found, activating TPM-bypass! [ 5.061363] ima: Allocated hash algorithm: sha1 [ 5.062744] ima: No architecture policies found [ 5.063977] evm: Initialising EVM extended attributes: [ 5.065879] evm: security.selinux [ 5.066985] evm: security.ima [ 5.067960] evm: security.capability [ 5.069256] evm: HMAC attrs: 0x1 [ 5.071745] rtc_cmos 00:05: setting system clock to 2026-02-18 18:29:10 UTC (1771439350) [ 5.078772] debug: unmapping init [mem 0xffffffff9a803000-0xffffffff9a9fffff] [ 5.081953] debug: unmapping init [mem 0xffffffff99582000-0xffffffff99858fff] [ 5.092121] Write protecting the kernel read-only data: 28672k [ 5.098392] debug: unmapping init [mem 0xffffffff97c03000-0xffffffff97dfffff] [ 5.102926] debug: unmapping init [mem 0xffffffff98514000-0xffffffff985fffff] [ 5.191243] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.212311] systemd[1]: Detected virtualization kvm. [ 5.217290] systemd[1]: Detected architecture x86-64. [ 5.219972] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.258969] systemd[1]: No hostname configured. [ 5.260464] systemd[1]: Set hostname to . [ 5.262270] random: systemd: uninitialized urandom read (16 bytes read) [ 5.264618] systemd[1]: Initializing machine ID from random generator. [ 5.365947] random: ln: uninitialized urandom read (6 bytes read) [ 5.617776] random: systemd: uninitialized urandom read (16 bytes read) [ 5.620398] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.632352] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 5.641585] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.503968] device-mapper: uevent: version 1.0.3 [ 6.506783] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 7.833308] virtio_net virtio0 ens2: renamed from eth0 [ 7.864174] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.013174] scsi host0: ata_piix [ 8.089897] scsi host1: ata_piix [ 8.093046] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.099224] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.589187] random: crng init done [ 13.597021] random: 7 urandom warning(s) missed due to ratelimiting [ 15.881301] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.414703] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ 18.996259] hrtimer: interrupt took 3763446 ns [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.548274] printk: systemd: 18 output lines suppressed due to ratelimiting [ 20.318323] SELinux: Disabled at runtime. [ 20.410038] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.420525] systemd[1]: Detected virtualization kvm. [ 20.422956] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.474475] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.478864] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.493520] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.499309] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.506334] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.516785] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.523992] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ 22.665267] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 23.964640] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.836503] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.915630] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.021795] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.145902] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit)[ 31.853082] Key type dns_resolver registered [ **] A start job is running for Configur…-only root support (9s / no limit)[ 32.358523] NFS: Registering the id_resolver key type [ 32.362394] Key type id_resolver registered [ 32.364578] Key type id_legacy registered [ *] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg230-server login: [ 64.946704] libcfs: loading out-of-tree module taints kernel. [ 64.965116] Key type ._llcrypt registered [ 64.966216] Key type .llcrypt registered [ 65.019868] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 73.152970] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 73.805881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.819386] alg: No test for adler32 (adler32-zlib) [ 74.903118] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 75.269633] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 76.911171] Key type lgssc registered [ 77.514888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.035936] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.944844] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 109.004345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 109.020755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 110.153239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 110.169091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 110.212260] Lustre: lustre-MDT0000: new disk, initializing [ 110.252805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 110.265519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 111.966445] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.081475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 118.120349] Lustre: 6481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 118.134446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 118.137407] Lustre: Skipped 1 previous similar message [ 118.180161] Lustre: lustre-MDT0001: new disk, initializing [ 118.208897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 118.224673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 118.231546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 119.757370] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.461932] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 126.482706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 126.593343] Lustre: lustre-OST0000: new disk, initializing [ 126.595184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 126.614413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.078099] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.529576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 129.534218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 129.571231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 135.162066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 135.223213] Lustre: lustre-OST0001: new disk, initializing [ 135.225825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 135.254109] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 137.289673] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.299412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 141.302771] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 141.327589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 144.732322] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 150.297179] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 157.919146] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing check_logdir /tmp/testlogs/ [ 160.441922] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing yml_node [ 162.505493] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 163.754099] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 164.987665] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 165.896788] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Feb 18 13:31:49 EST 2026 [ 173.194346] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 173.769066] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 187.360072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.360417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 187.365011] Lustre: Skipped 2 previous similar messages [ 187.369172] Lustre: Skipped 3 previous similar messages [ 192.155959] Lustre: server umount lustre-MDT0000 complete [ 193.498700] LustreError: 8381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771439538 with bad export cookie 4755607143597862040 [ 193.500097] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 193.504559] LustreError: 8381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 197.599423] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 197.599607] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 197.604829] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 197.611256] Lustre: Skipped 1 previous similar message [ 197.612698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 197.614783] Lustre: Skipped 1 previous similar message [ 198.623463] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 198.623498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 198.628372] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 198.632592] Lustre: Skipped 1 previous similar message [ 199.653400] Lustre: server umount lustre-MDT0001 complete [ 201.004752] Lustre: server umount lustre-OST0000 complete [ 202.353577] Lustre: server umount lustre-OST0001 complete [ 204.485311] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 207.554328] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 220.393040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.499120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 220.513837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 220.562530] Lustre: lustre-MDT0000: new disk, initializing [ 220.593393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.601814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.962619] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 226.273026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.319922] Lustre: 14942:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.435344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.838721] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 231.149567] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 232.312586] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 234.573558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.679705] Lustre: lustre-OST0000: new disk, initializing [ 234.681109] Lustre: Skipped 1 previous similar message [ 234.683434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.685791] Lustre: Skipped 2 previous similar messages [ 234.712375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.714751] Lustre: Skipped 1 previous similar message [ 236.284437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.287574] Lustre: Skipped 1 previous similar message [ 236.290505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.325961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.601351] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 239.766045] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 241.213378] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 241.293840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 242.698329] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 242.773972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 245.523736] Lustre: server umount lustre-OST0000 complete [ 247.263878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 247.266766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 247.271163] Lustre: Skipped 1 previous similar message [ 247.273921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 251.872175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 251.872952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.087122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 261.150225] Lustre: server umount lustre-MDT0000 complete [ 262.113121] LustreError: 15989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.120961] LustreError: 15989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 262.418470] LustreError: 15730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771439607 with bad export cookie 4755607143597863559 [ 262.421775] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 262.423911] LustreError: 15730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.900654] Lustre: server umount lustre-MDT0001 complete [ 273.336948] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:33:37 (1771439617) [ 274.173925] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 274.819220] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:33:38 (1771439618) [ 277.332793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.491339] LustreError: 18943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.513252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.793262] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 281.684792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.090268] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 284.890053] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 286.250300] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 288.851704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 291.096579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 293.042303] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 307.682349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 307.682627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.683673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 307.683678] Lustre: Skipped 3 previous similar messages [ 307.685055] LustreError: Skipped 1 previous similar message [ 307.694242] Lustre: Skipped 1 previous similar message [ 312.656702] Lustre: server umount lustre-OST0000 complete [ 317.920663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 317.920674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.927794] Lustre: Skipped 1 previous similar message [ 320.088889] Lustre: server umount lustre-MDT0000 complete [ 321.442769] LustreError: 21409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771439666 with bad export cookie 4755607143597864441 [ 321.446049] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.448148] LustreError: 21409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.577936] Lustre: server umount lustre-MDT0001 complete [ 325.245452] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 326.331755] Key type lgssc unregistered [ 326.456473] LNet: 22142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.458767] LNetError: 22142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.467372] LNet: Removed LNI 192.168.202.130@tcp [ 326.814136] Key type .llcrypt unregistered [ 326.815503] Key type ._llcrypt unregistered [ 335.433134] Key type ._llcrypt registered [ 335.434144] Key type .llcrypt registered [ 335.469884] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 335.795412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 335.833172] alg: No test for adler32 (adler32-zlib) [ 336.698600] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 336.782651] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 338.367157] Key type lgssc registered [ 338.733524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 341.348804] loop: module loaded [ 341.371814] loop0: detected capacity change from 0 to 204800000 [ 341.508476] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.511518] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.625085] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.596809] loop0: detected capacity change from 0 to 204800000 [ 343.646593] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.650815] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 343.654466] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 345.325265] loop0: detected capacity change from 0 to 204800000 [ 347.173065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 347.179924] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.282047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 348.296567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 348.331773] Lustre: lustre-MDT0000: new disk, initializing [ 348.356602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 348.365126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 349.646827] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 353.377872] loop1: detected capacity change from 0 to 204800000 [ 355.213394] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 355.310689] Lustre: lustre-OST07c6: new disk, initializing [ 355.312564] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 355.314799] Lustre: Skipped 1 previous similar message [ 355.338127] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 357.287959] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 363.006906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 363.009638] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 363.018946] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 365.727158] Lustre: 24275:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 365.730322] Lustre: 24275:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 365.733276] Lustre: 24275:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 365.736188] Lustre: 24275:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 365.740174] Lustre: 24275:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 365.743730] Lustre: 24275:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 381.077221] Lustre: 25279:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 381.081453] Lustre: 25279:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 381.084945] Lustre: 25279:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 381.087483] Lustre: 25279:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 381.091209] Lustre: 25279:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 381.094649] Lustre: 25279:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 384.608182] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 384.612038] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 384.615088] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 384.618435] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.621673] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 384.625544] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.627746] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 384.630885] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.633336] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 384.635902] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.638520] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 384.641080] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.697200] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 387.700769] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 387.702777] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 387.704772] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.706959] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 387.708802] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.712781] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 387.715034] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.718585] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 387.722013] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 387.725624] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 387.728537] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 394.237636] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 394.240825] Lustre: 24277:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 3 previous similar messages [ 394.243090] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 394.245201] Lustre: 24277:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 394.247927] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 394.250436] Lustre: 24277:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 394.253512] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 394.256610] Lustre: 24277:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 394.259910] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 394.262937] Lustre: 24277:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 394.266090] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 394.268976] Lustre: 24277:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 404.285527] Lustre: 25279:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 404.288088] Lustre: 25279:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 404.291304] Lustre: 25279:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 404.293378] Lustre: 25279:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 404.295396] Lustre: 25279:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 404.297498] Lustre: 25279:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 404.299462] Lustre: 25279:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 404.301658] Lustre: 25279:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 404.303733] Lustre: 25279:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 404.305841] Lustre: 25279:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 404.308153] Lustre: 25279:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 404.310519] Lustre: 25279:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 450.016412] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.017039] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.023233] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 451.147812] Lustre: server umount lustre-OST07c6 complete [ 454.234525] Lustre: server umount lustre-MDT0000 complete [ 460.005859] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 460.979951] Key type lgssc unregistered [ 461.098421] LNet: 27813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.101043] LNetError: 27813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 461.108925] LNet: Removed LNI 192.168.202.130@tcp [ 461.359928] Key type .llcrypt unregistered [ 461.360908] Key type ._llcrypt unregistered [ 468.176211] Key type ._llcrypt registered [ 468.177297] Key type .llcrypt registered [ 468.212032] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 468.530720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 468.604191] alg: No test for adler32 (adler32-zlib) [ 469.448243] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 469.530559] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 471.111122] Key type lgssc registered [ 471.447713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 473.874328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 473.880710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.033563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.331710] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 479.138923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.255828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 480.485199] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 482.226529] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 483.520037] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 485.827150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 485.924670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.676761] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 489.386086] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 500.261995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 509.407899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.410563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.415256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.433268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.434069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.434474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.334601] Lustre: server umount lustre-OST0000 complete [ 520.159681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 520.161860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.165676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.167399] Lustre: Skipped 1 previous similar message [ 522.708033] Lustre: server umount lustre-MDT0000 complete [ 523.895147] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771439869 with bad export cookie 10553821605765803292 [ 523.897944] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 523.898147] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.002813] Lustre: server umount lustre-MDT0001 complete [ 527.538951] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 528.553846] Key type lgssc unregistered [ 528.688534] LNet: 32444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.691397] LNetError: 32444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.704384] LNet: Removed LNI 192.168.202.130@tcp [ 528.986537] Key type .llcrypt unregistered [ 528.987814] Key type ._llcrypt unregistered [ 536.131985] Key type ._llcrypt registered [ 536.132985] Key type .llcrypt registered [ 536.164946] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 536.480931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 536.542226] alg: No test for adler32 (adler32-zlib) [ 537.398761] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 537.477931] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 539.056125] Key type lgssc registered [ 539.399334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 542.277189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 542.284692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.470121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.741462] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 547.539843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.651446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 548.911699] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 550.564850] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 551.688728] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 554.015741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 554.158895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.031793] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 557.799267] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 568.485719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 579.043035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.043701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.046540] LustreError: Skipped 1 previous similar message [ 579.051608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 579.054883] Lustre: Skipped 1 previous similar message [ 583.314313] Lustre: server umount lustre-OST0000 complete [ 588.767926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.770570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.774615] Lustre: Skipped 1 previous similar message [ 588.776649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.622940] Lustre: server umount lustre-MDT0000 complete [ 591.914502] LustreError: 35294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771439937 with bad export cookie 17618322669334798804 [ 591.917471] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.919025] LustreError: 35294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 592.026938] Lustre: server umount lustre-MDT0001 complete [ 595.655669] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 596.784890] Key type lgssc unregistered [ 596.915390] LNet: 36881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.918223] LNetError: 36881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.928348] LNet: Removed LNI 192.168.202.130@tcp [ 597.223677] Key type .llcrypt unregistered [ 597.225185] Key type ._llcrypt unregistered [ 606.139566] Key type ._llcrypt registered [ 606.140794] Key type .llcrypt registered [ 606.174866] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 606.499707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 606.544090] alg: No test for adler32 (adler32-zlib) [ 607.394958] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 607.479285] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 609.063216] Key type lgssc registered [ 609.412764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 611.929358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 611.935331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.082381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.346980] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 616.961324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.064024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 618.306366] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 620.191067] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 621.391456] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 623.686823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 623.798996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.650083] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 627.459361] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 639.144692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 647.648719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 647.651690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.657700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.294626] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 649.698924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.699484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.704325] Lustre: Skipped 1 previous similar message [ 649.706038] Lustre: Skipped 1 previous similar message [ 651.551271] Lustre: server umount lustre-MDT0000 complete [ 652.927463] LustreError: 39870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771439998 with bad export cookie 9089209854521793382 [ 652.930283] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.931690] LustreError: 39870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 653.406216] LustreError: 38252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.416414] LustreError: 38252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 653.416985] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 653.422329] Lustre: Skipped 1 previous similar message [ 654.816064] LustreError: 38280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.816349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.823796] LustreError: 38280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.824095] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 657.831877] Lustre: Skipped 1 previous similar message [ 659.102484] Lustre: server umount lustre-MDT0001 complete [ 685.023127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 685.072356] Lustre: server umount lustre-OST0000 complete [ 690.164778] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 691.205423] Key type lgssc unregistered [ 691.329319] LNet: 41782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.331462] LNetError: 41782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.338264] LNet: Removed LNI 192.168.202.130@tcp [ 691.612594] Key type .llcrypt unregistered [ 691.613672] Key type ._llcrypt unregistered [ 700.251428] Key type ._llcrypt registered [ 700.252561] Key type .llcrypt registered [ 700.288183] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 700.593646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.644078] alg: No test for adler32 (adler32-zlib) [ 701.501372] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 701.579564] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 703.159239] Key type lgssc registered [ 703.567311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 706.369121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 706.376607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.533963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.939425] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 711.969993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.107153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 713.448871] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 715.209265] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 716.429843] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 718.582742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.708371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.582477] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 722.373264] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 732.006749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 737.249597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 737.250143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.252242] LustreError: Skipped 1 previous similar message [ 737.252904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 737.257143] Lustre: Skipped 1 previous similar message [ 737.261896] Lustre: Skipped 1 previous similar message [ 737.892256] Lustre: lustre-OST0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 741.006921] Lustre: server umount lustre-OST0000 complete [ 753.771632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.777288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.832779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.832818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.957097] Lustre: server umount lustre-MDT0000 complete [ 761.124084] LustreError: 43138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440106 with bad export cookie 1464476609929920890 [ 761.126401] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.128970] LustreError: 43138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 767.284384] Lustre: server umount lustre-MDT0001 complete [ 770.965642] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 772.011512] Key type lgssc unregistered [ 772.147423] LNet: 46532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.150114] LNetError: 46532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.159333] LNet: Removed LNI 192.168.202.130@tcp [ 772.477467] Key type .llcrypt unregistered [ 772.478799] Key type ._llcrypt unregistered [ 781.549618] Key type ._llcrypt registered [ 781.551401] Key type .llcrypt registered [ 781.596468] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 782.095626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.102952] alg: No test for adler32 (adler32-zlib) [ 782.967665] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 783.062529] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 784.647144] Key type lgssc registered [ 785.041973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.671207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.679756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.857029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.091825] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 792.740729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.855379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.067991] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 795.769699] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 796.999391] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 799.117220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 799.230274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 801.111199] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 802.832811] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 803.366399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 810.936233] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 812.118605] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 822.813617] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-02-1813h42m46s [ 823.313595] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 824.417717] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 825.436159] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 828.897573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.897769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 828.899237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 828.901311] Lustre: Skipped 1 previous similar message [ 830.165755] Lustre: server umount lustre-MDT0000 complete [ 831.214928] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 831.579413] LustreError: 50061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 831.584141] LustreError: 50061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 833.301599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.342128] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 833.421185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.444159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 834.469673] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 834.590214] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 835.622812] Lustre: DEBUG MARKER: Wait for df (36017) ... [ 836.099429] Lustre: DEBUG MARKER: done [ 838.624059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.627675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.631984] Lustre: Skipped 1 previous similar message [ 838.633921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 838.635753] Lustre: Skipped 6 previous similar messages [ 843.086177] Lustre: server umount lustre-OST0000 complete [ 844.768858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.768884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 844.769529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.769532] Lustre: Skipped 2 previous similar messages [ 844.772901] Lustre: Skipped 2 previous similar messages [ 844.775035] LustreError: Skipped 1 previous similar message [ 850.320167] Lustre: server umount lustre-MDT0000 complete [ 850.848808] LustreError: 47926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.854933] LustreError: 47926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 851.518019] LustreError: 49523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440196 with bad export cookie 85040496422389422 [ 851.520319] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 851.522954] LustreError: 49523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 851.643995] Lustre: server umount lustre-MDT0001 complete [ 855.119180] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 856.125826] Key type lgssc unregistered [ 856.248420] LNet: 52456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.251416] LNetError: 52456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.264330] LNet: Removed LNI 192.168.202.130@tcp [ 856.546372] Key type .llcrypt unregistered [ 856.547269] Key type ._llcrypt unregistered [ 869.831185] Key type ._llcrypt registered [ 869.832283] Key type .llcrypt registered [ 869.864970] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 870.158952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 870.212386] alg: No test for adler32 (adler32-zlib) [ 871.061834] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 871.139593] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 872.719145] Key type lgssc registered [ 873.071535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 875.527639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 875.535177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.617031] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 876.624628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 876.678558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.829920] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 880.522770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.550673] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 880.559362] Lustre: 54624:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 880.649472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.933498] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 883.638651] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 884.724860] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 886.817244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 886.880713] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 886.938542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.714518] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 890.351558] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 893.967495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 898.688841] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-02-1813h44m02s [ 899.195700] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 901.313205] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 902.053791] Lustre: DEBUG MARKER: Creating a test file and stat it [ 902.066560] LustreError: 55853:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 922.216482] Lustre: lustre-MDT0000: Client 17127361-8dac-4d03-b7a1-073ad7cb2258 (at 192.168.202.30@tcp) reconnecting [ 942.111116] LustreError: 55853:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 942.874705] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 943.664296] Lustre: DEBUG MARKER: done [ 948.193028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 948.195570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.199457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 950.241429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.241698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 950.242739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 951.242574] Lustre: server umount lustre-OST0000 complete [ 955.360677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.361129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 955.364589] Lustre: Skipped 1 previous similar message [ 955.366586] Lustre: Skipped 1 previous similar message [ 958.550190] Lustre: server umount lustre-MDT0000 complete [ 959.765533] LustreError: 54613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440305 with bad export cookie 2572459429367898201 [ 959.768257] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 959.770151] LustreError: 54613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 959.900522] Lustre: server umount lustre-MDT0001 complete [ 963.247498] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 964.315692] Key type lgssc unregistered [ 964.451454] LNet: 58591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.454539] LNetError: 58591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.461300] LNet: Removed LNI 192.168.202.130@tcp [ 964.760781] Key type .llcrypt unregistered [ 964.762091] Key type ._llcrypt unregistered [ 980.094270] Key type ._llcrypt registered [ 980.095279] Key type .llcrypt registered [ 980.130100] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 980.609452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 980.622089] alg: No test for adler32 (adler32-zlib) [ 981.475452] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 981.555028] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 983.135160] Key type lgssc registered [ 983.481278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.524141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.529865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.627488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 985.636598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 985.700807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.058382] Lustre: Failing over lustre-MDT0000 [ 986.158065] Lustre: server umount lustre-MDT0000 complete [ 988.320220] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:45:32 (1771440332) [ 990.496044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.634495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.890348] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 994.754356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 994.790665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 994.803732] Lustre: 61349:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 994.901428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 996.295135] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 998.136787] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 999.304416] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1001.483475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.545966] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1001.601563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.295959] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1004.900573] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1006.601200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1013.170207] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1015.263914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1015.266279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.270128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.800630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.801917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.802496] Lustre: Skipped 1 previous similar message [ 1016.806220] Lustre: Skipped 1 previous similar message [ 1021.072512] Lustre: server umount lustre-MDT0000 complete [ 1022.237378] LustreError: 62067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440367 with bad export cookie 13341694400898271166 [ 1022.241251] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1022.241773] LustreError: 62067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.342630] Lustre: server umount lustre-MDT0001 complete [ 1023.380139] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1024.844436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1027.486395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.160203] LustreError: 60272:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f2119956 x1857489811307648/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1032.260218] LustreError: 62970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.265216] LustreError: 62970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1032.293416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1032.325528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1033.443144] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1036.093702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.189117] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.485765] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1039.311750] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1040.453776] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1041.380257] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1043.423161] Lustre: 60275:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771440372/real 1771440372] req@0000000080ef3118 x1857489811307392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771440388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1048.287188] Lustre: 60275:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771440377/real 1771440377] req@00000000a118f520 x1857489811307520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771440393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1048.289656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.290193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.307427] Lustre: Skipped 3 previous similar messages [ 1053.410087] Lustre: server umount lustre-MDT0000 complete [ 1053.664599] LustreError: 66036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.672486] LustreError: 66036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1054.670382] LustreError: 66088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440400 with bad export cookie 13341694400898273434 [ 1054.675763] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1054.680247] LustreError: 66088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1054.791180] Lustre: server umount lustre-MDT0001 complete [ 1056.936603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1060.213800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.927703] LustreError: 60272:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bbc0f1fd x1857489811327616/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1065.045579] LustreError: 64792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.046487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.061145] Lustre: Skipped 1 previous similar message [ 1065.079951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.082265] Lustre: Skipped 1 previous similar message [ 1065.105139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1066.434348] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1069.229379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.685218] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1072.404156] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1073.503045] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1074.661162] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1075.679399] Lustre: 60274:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771440405/real 1771440405] req@00000000b605f537 x1857489811327360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771440421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1079.775153] Lustre: 60275:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771440409/real 1771440409] req@000000002ff37a1f x1857489811327488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771440425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1079.784359] Lustre: 60275:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1084.384600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.385737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.391912] Lustre: Skipped 1 previous similar message [ 1086.423809] Lustre: server umount lustre-MDT0000 complete [ 1087.581121] LustreError: 61339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440433 with bad export cookie 13341694400898274806 [ 1087.584643] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1087.584975] LustreError: 61339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1087.680571] Lustre: server umount lustre-MDT0001 complete [ 1089.610651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1090.662592] Lustre: DEBUG MARKER: files compared the same [ 1101.604182] Lustre: server umount lustre-OST0000 complete [ 1106.678233] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1107.638818] Key type lgssc unregistered [ 1107.759380] LNet: 70260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.761385] LNetError: 70260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.768331] LNet: Removed LNI 192.168.202.130@tcp [ 1108.046764] Key type .llcrypt unregistered [ 1108.047859] Key type ._llcrypt unregistered [ 1116.369331] Key type ._llcrypt registered [ 1116.370401] Key type .llcrypt registered [ 1116.411860] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1116.911736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1116.923049] alg: No test for adler32 (adler32-zlib) [ 1117.777049] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1117.859206] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1119.439196] Key type lgssc registered [ 1119.814420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1122.508717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.514074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.684097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.039907] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug +malloc all [ 1127.950224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.058678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1129.425307] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug +malloc all [ 1131.269498] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.534124] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1134.807215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1134.900078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.715760] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug +malloc all [ 1138.342310] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1149.218514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1158.623848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.627165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.634143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.648254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.648507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.648770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.701855] Lustre: server umount lustre-OST0000 complete [ 1164.255610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1164.258329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.262468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.264497] Lustre: Skipped 1 previous similar message [ 1169.887923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.890962] Lustre: Skipped 2 previous similar messages [ 1170.072208] Lustre: server umount lustre-MDT0000 complete [ 1171.376603] LustreError: 71616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440516 with bad export cookie 12387418191737612134 [ 1171.378087] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.379959] LustreError: 71616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1171.489232] Lustre: server umount lustre-MDT0001 complete [ 1177.745977] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1179.241074] Key type lgssc unregistered [ 1179.410724] LNet: 74835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.415214] LNetError: 74835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.423244] LNet: Removed LNI 192.168.202.130@tcp [ 1180.269742] Key type .llcrypt unregistered [ 1180.271541] Key type ._llcrypt unregistered [ 1197.412408] Key type ._llcrypt registered [ 1197.421851] Key type .llcrypt registered [ 1197.528218] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1198.652701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.697403] alg: No test for adler32 (adler32-zlib) [ 1199.804499] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1200.149942] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1201.817058] Key type lgssc registered [ 1203.103186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.479408] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.521589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1227.232489] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1232.351186] LustreError: 76187:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 1258.975756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.049031] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1264.530448] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1268.779980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.884762] Lustre: *** cfs_fail_loc=706, val=0*** [ 1269.140580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.187404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1271.284681] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1274.275420] LustreError: 76212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.283683] LustreError: 76212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1275.500160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.698743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.635779] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1280.162223] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1282.103483] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1284.576712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1284.586921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.597166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.113702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.113782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.116601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.242037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1291.244505] Lustre: Skipped 2 previous similar messages [ 1296.354510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.357536] Lustre: Skipped 1 previous similar message [ 1297.375957] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1297.436216] Lustre: server umount lustre-OST0000 complete [ 1301.473022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.473961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.480724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.489462] Lustre: Skipped 1 previous similar message [ 1305.196283] Lustre: server umount lustre-MDT0000 complete [ 1306.592589] LustreError: 77162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.599222] LustreError: 77162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1306.700387] LustreError: 76194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440652 with bad export cookie 16920630931979519598 [ 1306.704040] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.705964] LustreError: 76194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1306.822929] Lustre: server umount lustre-MDT0001 complete [ 1310.949863] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1312.038137] Key type lgssc unregistered [ 1312.188363] LNet: 79449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.191552] LNetError: 79449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.199488] LNet: Removed LNI 192.168.202.130@tcp [ 1312.525869] Key type .llcrypt unregistered [ 1312.527750] Key type ._llcrypt unregistered [ 1324.296448] Key type ._llcrypt registered [ 1324.298122] Key type .llcrypt registered [ 1324.344360] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1324.693385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.764527] alg: No test for adler32 (adler32-zlib) [ 1325.668590] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1325.803535] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1327.431959] Key type lgssc registered [ 1328.048968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.320583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.328272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.836116] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1337.075834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.350703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.989719] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1342.434348] LustreError: 81493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.443696] LustreError: 81493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1343.555864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.760129] LustreError: 81493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.768160] LustreError: 81493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1343.770744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.680239] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1346.858451] LustreError: 82222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.869309] LustreError: 82222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1351.308645] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1351.315427] Lustre: MGS: Client 84ee93fa-bd67-4dee-a5e2-8f3884f85629 (at 0@lo) reconnecting [ 1351.329935] LustreError: 81507:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1351.453954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.494797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1353.694048] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1367.010172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.012652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.014258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.020881] Lustre: Skipped 1 previous similar message [ 1367.025819] Lustre: Skipped 1 previous similar message [ 1369.619593] Lustre: server umount lustre-OST0000 complete [ 1372.128873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.129974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1372.130100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.135118] Lustre: Skipped 1 previous similar message [ 1377.251992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.254644] Lustre: Skipped 2 previous similar messages [ 1377.377420] Lustre: server umount lustre-MDT0000 complete [ 1379.572520] Lustre: server umount lustre-MDT0001 complete [ 1381.644774] Lustre: server umount MGS complete [ 1387.995670] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:52:11 (1771440731) [ 1404.544774] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 1409.631141] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1428.588227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.840981] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1436.971101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.098487] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1437.124147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1437.166741] Lustre: lustre-MDT0001: new disk, initializing [ 1437.211069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1437.222366] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1437.225453] Lustre: 88555:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1437.231087] LustreError: 88555:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1439.515650] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1445.671974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1445.815285] Lustre: lustre-OST0000: new disk, initializing [ 1445.818158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1445.865154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.034037] Lustre: 89432:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b56285c5 x1857490173070080/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1771440809 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1448.975850] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1453.087574] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1456.674635] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.685661] Lustre: MGS: Client a6f361ef-ca9a-4186-a067-b11baa58fbde (at 0@lo) reconnecting [ 1456.711055] Lustre: 90234:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1456.724599] LustreError: 90234:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1456.728696] LustreError: 90234:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1456.745945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1456.798176] Lustre: lustre-MDT0000: new disk, initializing [ 1456.892544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.907345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1457.120085] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1457.954421] Lustre: 90261:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000007d91b932 x1857490173076224/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1771440819 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1458.919886] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1467.044042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1467.059450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1467.117227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1489.889253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.897640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1489.901527] Lustre: Skipped 1 previous similar message [ 1495.011493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.018578] Lustre: Skipped 1 previous similar message [ 1504.223246] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1504.309928] Lustre: server umount lustre-OST0000 complete [ 1510.371072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1510.372452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.374766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1510.374771] Lustre: Skipped 2 previous similar messages [ 1510.391280] Lustre: Skipped 1 previous similar message [ 1512.404892] Lustre: server umount lustre-MDT0000 complete [ 1520.706780] Lustre: server umount lustre-MDT0001 complete [ 1522.943802] Lustre: server umount MGS complete [ 1528.812247] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:54:32 (1771440872) [ 1537.929595] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1539.637349] Key type lgssc unregistered [ 1539.819930] LNet: 92726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.823930] LNetError: 92726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.836866] LNet: Removed LNI 192.168.202.130@tcp [ 1540.303661] Key type .llcrypt unregistered [ 1540.308639] Key type ._llcrypt unregistered [ 1551.126565] Key type ._llcrypt registered [ 1551.128945] Key type .llcrypt registered [ 1551.197467] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1551.953707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1551.971567] alg: No test for adler32 (adler32-zlib) [ 1552.899459] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1553.085274] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1554.743173] Key type lgssc registered [ 1555.476562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.281202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1560.293802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.576453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.716345] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1568.373308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.615665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1570.955598] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1574.080942] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1576.387526] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1580.527867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1580.767180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.270244] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1585.897245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1587.885907] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1598.433967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.434049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.438762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.449592] Lustre: Skipped 1 previous similar message [ 1600.095511] Lustre: server umount lustre-OST0000 complete [ 1603.554716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.557764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.566903] Lustre: Skipped 1 previous similar message [ 1603.577458] Lustre: Skipped 2 previous similar messages [ 1608.674258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.677247] Lustre: Skipped 1 previous similar message [ 1608.816969] Lustre: server umount lustre-MDT0000 complete [ 1610.879171] LustreError: 93889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771440956 with bad export cookie 6372149751427645509 [ 1610.880239] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1610.889324] LustreError: 93889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1611.087298] Lustre: server umount lustre-MDT0001 complete [ 1617.337150] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1619.054729] Key type lgssc unregistered [ 1619.272251] LNet: 97086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.276590] LNetError: 97086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.290633] LNet: Removed LNI 192.168.202.130@tcp [ 1619.820635] Key type .llcrypt unregistered [ 1619.822616] Key type ._llcrypt unregistered [ 1629.026960] Key type ._llcrypt registered [ 1629.028312] Key type .llcrypt registered [ 1629.089962] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1629.681581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1629.706408] alg: No test for adler32 (adler32-zlib) [ 1630.645313] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1630.786204] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1632.415199] Key type lgssc registered [ 1633.160289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.764495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.768568] LustreError: 98132:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1636.775538] LustreError: 98133:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1636.778453] LustreError: 98132:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1636.782251] LustreError: 98132:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1636.787475] LustreError: 98132:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1636.794154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.092141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.213037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.402881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1646.267196] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1649.924185] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing lsmod [ 1651.905664] LustreError: 99303:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 1651.905633] LustreError: 99304:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 1651.918308] LustreError: 99304:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 1651.927652] LustreError: 99304:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1651.932398] LustreError: 99304:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1651.939044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1652.164995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.341908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.344340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.351753] Lustre: Skipped 2 previous similar messages [ 1658.359152] Lustre: Skipped 2 previous similar messages [ 1660.064174] Lustre: server umount lustre-MDT0000 complete [ 1662.432856] LustreError: 98150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441007 with bad export cookie 5652406037818984001 [ 1662.439186] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1662.445345] LustreError: 98150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1662.664748] Lustre: server umount lustre-MDT0001 complete [ 1675.212777] Lustre: server umount lustre-OST0000 complete [ 1680.117345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.372482] LustreError: 100372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.383949] LustreError: 100372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1680.416456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.945609] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1685.472676] LustreError: 100372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.765540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.943392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.640775] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1692.127614] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1693.800412] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1697.144223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1697.300917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.247404] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1702.995564] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1705.514373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1713.633373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1713.643293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.657215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.680190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1715.683935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.698458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.704223] Lustre: Skipped 1 previous similar message [ 1717.665657] Lustre: server umount lustre-OST0000 complete [ 1720.801407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.802369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.808158] Lustre: Skipped 1 previous similar message [ 1725.920299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.926488] Lustre: Skipped 2 previous similar messages [ 1734.111177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1734.190463] Lustre: server umount lustre-MDT0000 complete [ 1736.162296] LustreError: 100372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.180622] LustreError: 100372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1736.501456] LustreError: 100352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441081 with bad export cookie 5652406037818984708 [ 1736.509492] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.510697] LustreError: 100352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1736.772900] Lustre: server umount lustre-MDT0001 complete [ 1743.173615] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1744.891740] Key type lgssc unregistered [ 1745.066826] LNet: 103547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.072744] LNetError: 103547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.088982] LNet: Removed LNI 192.168.202.130@tcp [ 1745.578638] Key type .llcrypt unregistered [ 1745.580645] Key type ._llcrypt unregistered [ 1757.879099] Key type ._llcrypt registered [ 1757.880343] Key type .llcrypt registered [ 1757.925953] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1758.476386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1758.493449] alg: No test for adler32 (adler32-zlib) [ 1759.413380] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1759.516773] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1761.127256] Key type lgssc registered [ 1761.742184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1765.739465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1765.747903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.003547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.021599] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1773.633316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.870186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1776.131653] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1778.782919] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1780.656619] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1784.227581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1784.384820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.368514] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1790.641522] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1791.595424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1803.624458] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1805.955754] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1809.383596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.388151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.391980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.405799] Lustre: Skipped 1 previous similar message [ 1814.394985] Lustre: server umount lustre-OST0000 complete [ 1819.618051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.625488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1819.629615] Lustre: Skipped 2 previous similar messages [ 1819.633888] Lustre: Skipped 1 previous similar message [ 1822.407477] Lustre: server umount lustre-MDT0000 complete [ 1824.539450] LustreError: 104904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441169 with bad export cookie 5928787869230700843 [ 1824.541049] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1824.553801] LustreError: 104904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1824.733368] Lustre: server umount lustre-MDT0001 complete [ 1830.953478] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1832.619897] Key type lgssc unregistered [ 1832.817198] LNet: 108222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.823483] LNetError: 108222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.838470] LNet: Removed LNI 192.168.202.130@tcp [ 1833.374940] Key type .llcrypt unregistered [ 1833.377129] Key type ._llcrypt unregistered [ 1843.740771] Key type ._llcrypt registered [ 1843.742578] Key type .llcrypt registered [ 1843.809661] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1844.419681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1844.475813] alg: No test for adler32 (adler32-zlib) [ 1845.430451] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1845.529563] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1847.143179] Key type lgssc registered [ 1847.824069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1852.322395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1852.337682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.628878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.902199] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1861.136031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.394178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.681274] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1867.021508] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1869.367772] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1873.781473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.050516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.456851] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1879.142834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1880.674370] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1891.810357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.814268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.820970] Lustre: Skipped 1 previous similar message [ 1891.823579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.833960] Lustre: Skipped 1 previous similar message [ 1892.179238] Lustre: server umount lustre-OST0000 complete [ 1896.932433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.943969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.913495] Lustre: server umount lustre-MDT0000 complete [ 1902.050296] LustreError: 109408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.058109] LustreError: 109408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1902.944066] LustreError: 109389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441248 with bad export cookie 14873606728935690135 [ 1902.944858] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.955245] LustreError: 109389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1903.142693] Lustre: server umount lustre-MDT0001 complete [ 1908.648721] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1910.145249] Key type lgssc unregistered [ 1910.332665] LNet: 112583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.335909] LNetError: 112583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.346733] LNet: Removed LNI 192.168.202.130@tcp [ 1910.773979] Key type .llcrypt unregistered [ 1910.775568] Key type ._llcrypt unregistered [ 1923.153439] Key type ._llcrypt registered [ 1923.154735] Key type .llcrypt registered [ 1923.202634] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1923.690331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1923.757110] alg: No test for adler32 (adler32-zlib) [ 1924.676443] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 1924.804168] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1926.391184] Key type lgssc registered [ 1927.009534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.303655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.316885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.613437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.771096] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1939.695574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.860614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1941.857346] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1944.475633] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1946.337787] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1949.989098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1950.122059] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1950.201721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.488334] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1956.868143] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1958.382090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1966.738804] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1968.703817] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1968.709374] Lustre: Skipped 2 previous similar messages [ 1970.868863] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1970.872301] Lustre: Skipped 2 previous similar messages [ 1975.830978] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1975.852550] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1980.090977] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1980.096229] Lustre: Skipped 2 previous similar messages [ 1983.526274] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.30@tcp 192.168.0.[2,10]@tcp [ 1985.176529] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.30/24@tcp in log lustre-MDT0000 [ 1985.179091] Lustre: Skipped 2 previous similar messages [ 1989.284627] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.30/24@tcp [ 1989.290086] Lustre: Skipped 1 previous similar message [ 1994.213105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.213973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.213983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.223288] Lustre: Skipped 1 previous similar message [ 1998.204520] Lustre: server umount lustre-OST0000 complete [ 2001.375952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2001.382668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.389203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.394853] Lustre: Skipped 1 previous similar message [ 2004.453186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.453247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.470365] Lustre: Skipped 1 previous similar message [ 2006.248888] Lustre: server umount lustre-MDT0000 complete [ 2007.987321] LustreError: 113971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441353 with bad export cookie 8954243622844956693 [ 2007.989782] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.992027] LustreError: 113971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2008.138027] Lustre: server umount lustre-MDT0001 complete [ 2012.328766] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2013.439796] Key type lgssc unregistered [ 2013.587399] LNet: 118617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.590424] LNetError: 118617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.598694] LNet: Removed LNI 192.168.202.130@tcp [ 2013.926110] Key type .llcrypt unregistered [ 2013.927441] Key type ._llcrypt unregistered [ 2026.642789] Key type ._llcrypt registered [ 2026.643951] Key type .llcrypt registered [ 2026.684080] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2027.171809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.182284] alg: No test for adler32 (adler32-zlib) [ 2028.056338] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 2028.151392] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2029.751154] Key type lgssc registered [ 2030.187363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.711488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.720810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.953187] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2034.962593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.966504] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.30/24@tcp [ 2036.571396] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2040.162633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.336663] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2040.349457] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.30/24@tcp [ 2040.377953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2041.976346] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2044.102816] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2045.644149] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2048.600658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.771423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.377214] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2053.883767] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2061.093449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2070.495960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2070.500510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.508865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.522278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2071.525556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.526130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.537040] Lustre: Skipped 1 previous similar message [ 2076.219454] Lustre: server umount lustre-OST0000 complete [ 2081.248965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2081.252443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.259504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.389296] Lustre: server umount lustre-MDT0000 complete [ 2086.390955] LustreError: 120120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441431 with bad export cookie 15151380850226757599 [ 2086.399493] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2086.608986] Lustre: server umount lustre-MDT0001 complete [ 2092.160770] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2093.611407] Key type lgssc unregistered [ 2093.789435] LNet: 123390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.796607] LNetError: 123390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.807454] LNet: Removed LNI 192.168.202.130@tcp [ 2094.296927] Key type .llcrypt unregistered [ 2094.301965] Key type ._llcrypt unregistered [ 2107.065547] Key type ._llcrypt registered [ 2107.068812] Key type .llcrypt registered [ 2107.127345] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2114.562696] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2114.943112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2114.992933] alg: No test for adler32 (adler32-zlib) [ 2115.882551] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 2115.989637] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2117.575174] Key type lgssc registered [ 2118.042878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.781634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.790775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.935168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2133.948820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2133.989879] Lustre: lustre-MDT0000: new disk, initializing [ 2134.026627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.034662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2135.714776] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2141.010400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.059241] Lustre: 126731:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2141.088958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2141.091704] Lustre: Skipped 1 previous similar message [ 2141.131884] Lustre: lustre-MDT0001: new disk, initializing [ 2141.180126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.202396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2141.209725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2142.912591] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2146.774204] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2148.610355] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2151.835644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2151.965627] Lustre: lustre-OST0000: new disk, initializing [ 2151.969386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2152.035572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.317617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2153.326982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2153.375260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2154.910140] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2158.871676] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2161.047095] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2161.148758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2163.240535] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2163.376402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2168.802853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.806043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2168.807030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.814340] Lustre: Skipped 1 previous similar message [ 2170.718700] Lustre: server umount lustre-OST0000 complete [ 2173.920621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.923461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.926887] Lustre: Skipped 1 previous similar message [ 2173.929344] Lustre: Skipped 1 previous similar message [ 2179.048177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.051186] Lustre: Skipped 2 previous similar messages [ 2179.201798] Lustre: server umount lustre-MDT0000 complete [ 2181.308920] LustreError: 126724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441526 with bad export cookie 13388474300208671023 [ 2181.312569] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.314103] LustreError: 126724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2181.513037] Lustre: server umount lustre-MDT0001 complete [ 2186.625533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.907471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.243752] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2192.354111] LustreError: 130540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.369172] LustreError: 130540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2193.773957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.993107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.239210] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2199.500791] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2201.653576] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2205.331621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.510191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.314985] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2210.801411] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2212.597304] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2212.694454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2214.505239] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2219.722273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 5 sec [ 2225.845716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2225.921381] Lustre: lustre-OST0001: new disk, initializing [ 2225.923406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2225.968930] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2228.638283] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2233.099713] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2235.404302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2235.410556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2235.453574] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2236.261778] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2237.925355] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2238.020820] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2239.696874] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b1ad90fd000.ost_server_uuid 50 [ 2240.418271] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b1ad90fd000.ost_server_uuid in FULL state after 0 sec [ 2245.605337] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.607393] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2245.615245] Lustre: Skipped 1 previous similar message [ 2245.616167] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2245.621981] LustreError: Skipped 1 previous similar message [ 2248.976738] Lustre: server umount lustre-OST0001 complete [ 2250.727991] LustreError: 133190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.738561] LustreError: 133190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2252.768498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.775077] LustreError: Skipped 1 previous similar message [ 2252.777602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.782829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.787954] Lustre: Skipped 1 previous similar message [ 2254.816054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2255.845120] LustreError: 132132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.851885] LustreError: 132132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2256.982531] Lustre: server umount lustre-OST0000 complete [ 2260.448340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2260.452119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.456223] Lustre: Skipped 1 previous similar message [ 2260.961632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.965570] Lustre: Skipped 4 previous similar messages [ 2264.552119] Lustre: server umount lustre-MDT0000 complete [ 2266.080733] LustreError: 130541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.091978] LustreError: 130541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2266.154602] LustreError: 130520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441611 with bad export cookie 13388474300208671891 [ 2266.156661] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.160564] LustreError: 130520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2266.321473] Lustre: server umount lustre-MDT0001 complete [ 2274.601169] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2276.035235] Key type lgssc unregistered [ 2276.198649] LNet: 136431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.202399] LNetError: 136431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.212958] LNet: Removed LNI 192.168.202.130@tcp [ 2276.577499] Key type .llcrypt unregistered [ 2276.579431] Key type ._llcrypt unregistered [ 2296.143225] Key type ._llcrypt registered [ 2296.145376] Key type .llcrypt registered [ 2296.223566] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2303.825851] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2304.400738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2304.412808] alg: No test for adler32 (adler32-zlib) [ 2305.302344] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 2305.412177] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2307.015182] Key type lgssc registered [ 2307.495503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2324.869289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2324.877443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.992891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2326.010362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2326.052410] Lustre: lustre-MDT0000: new disk, initializing [ 2326.088704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.100939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2327.761564] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2333.137766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.195688] Lustre: 140526:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2333.226502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2333.229499] Lustre: Skipped 1 previous similar message [ 2333.268531] Lustre: lustre-MDT0001: new disk, initializing [ 2333.317890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.334666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2333.338957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2335.300813] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2339.650336] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2341.469912] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2344.997538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.113094] Lustre: lustre-OST0000: new disk, initializing [ 2345.115750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2345.149292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.476201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2347.479793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2347.534943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2347.556606] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2351.291691] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2354.633810] Lustre: Failing over lustre-OST0000 [ 2354.717139] Lustre: server umount lustre-OST0000 complete [ 2368.714776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.863125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.868520] Lustre: Skipped 1 previous similar message [ 2368.879419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.886843] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2369.133995] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2371.737661] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2373.391893] Lustre: Failing over lustre-MDT0000 [ 2373.530606] Lustre: server umount lustre-MDT0000 complete [ 2374.111130] Lustre: 138753:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771441703/real 1771441703] req@000000001265fc76 x1857491199618048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1771441719 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2374.114584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.129724] Lustre: Skipped 1 previous similar message [ 2379.232347] LustreError: 140536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.240707] LustreError: 140536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2379.551154] Lustre: 138752:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771441708/real 1771441708] req@0000000059c63be5 x1857491199619840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771441724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2383.647122] Lustre: 138752:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771441713/real 1771441713] req@00000000814e9e82 x1857491199620992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771441729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2384.352087] LustreError: 140536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.360497] LustreError: 140536:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2387.573299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.630421] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2387.638153] Lustre: 140528:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.202.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2387.733350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2387.763491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.764486] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2387.764723] Lustre: lustre-OST0000: Recovery over after 0:18, of 3 clients 3 recovered and 0 were evicted. [ 2387.797761] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2389.597271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2389.751484] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2393.064706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2393.067109] Lustre: Skipped 1 previous similar message [ 2393.072479] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2393.094864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2398.186102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2398.188960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.197058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.205424] Lustre: Skipped 2 previous similar messages [ 2398.221625] Lustre: Skipped 1 previous similar message [ 2400.448655] Lustre: server umount lustre-OST0000 complete [ 2403.298721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2403.299471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.300595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.303533] LustreError: Skipped 1 previous similar message [ 2403.316289] Lustre: Skipped 1 previous similar message [ 2408.291393] Lustre: server umount lustre-MDT0000 complete [ 2408.416371] LustreError: 141723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.423319] LustreError: 141723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2409.904290] LustreError: 140518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441755 with bad export cookie 7439361883547843771 [ 2409.907762] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.911342] LustreError: 140518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2410.055373] Lustre: server umount lustre-MDT0001 complete [ 2414.157603] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2415.359275] Key type lgssc unregistered [ 2415.502600] LNet: 145750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.505699] LNetError: 145750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.514312] LNet: Removed LNI 192.168.202.130@tcp [ 2415.872503] Key type .llcrypt unregistered [ 2415.873830] Key type ._llcrypt unregistered [ 2438.435926] Key type ._llcrypt registered [ 2438.437570] Key type .llcrypt registered [ 2438.491529] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2438.886551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.931756] alg: No test for adler32 (adler32-zlib) [ 2439.801467] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 2439.896905] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2441.487176] Key type lgssc registered [ 2442.034078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2445.499375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2445.506658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.700569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.478185] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2452.699862] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2455.292982] LustreError: 147297:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2455.306773] LustreError: 147297:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2455.313190] LustreError: 147297:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2455.530697] Lustre: server umount lustre-MDT0000 complete [ 2462.142389] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2465.959055] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2481.457461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.560453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2481.574202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2481.614909] Lustre: lustre-MDT0000: new disk, initializing [ 2481.646913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.654724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2483.185932] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2487.942790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.991602] Lustre: 150448:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2488.018712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2488.021026] Lustre: Skipped 1 previous similar message [ 2488.059326] Lustre: lustre-MDT0001: new disk, initializing [ 2488.107900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2488.128118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2488.134740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2489.692147] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2493.168922] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2494.615705] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2497.122904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.248958] Lustre: lustre-OST0000: new disk, initializing [ 2497.251694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2497.293089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.488223] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2502.793089] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2504.699653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2504.703553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2504.733687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2510.082321] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2510.680959] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:10:54 (1771441854) [ 2600.927407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.930462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.935485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.951391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.951561] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2603.353971] Lustre: server umount lustre-MDT0000 complete [ 2604.594659] LustreError: 150438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771441950 with bad export cookie 2231096596408348490 [ 2604.597701] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.599629] LustreError: 150438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2607.071394] LustreError: 152789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.071444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.076765] LustreError: 152789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2607.080780] Lustre: Skipped 1 previous similar message [ 2607.085821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2607.088545] Lustre: Skipped 2 previous similar messages [ 2609.119487] LustreError: 152789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.119523] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2610.787925] Lustre: server umount lustre-MDT0001 complete [ 2618.321721] Lustre: server umount lustre-OST0000 complete [ 2621.480912] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2624.713114] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2637.219990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.308492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2637.320674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2637.359114] Lustre: lustre-MDT0000: new disk, initializing [ 2637.385728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.393812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2638.767579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2643.324685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.368047] Lustre: 156672:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2643.401456] LustreError: 156682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.407878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2643.410962] Lustre: Skipped 1 previous similar message [ 2643.451489] Lustre: lustre-MDT0001: new disk, initializing [ 2643.512472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2643.516367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2645.020781] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2648.468922] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2649.858536] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2652.495508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.646762] Lustre: lustre-OST0000: new disk, initializing [ 2652.651482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2652.711761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.714687] Lustre: Skipped 1 previous similar message [ 2655.021053] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2657.286773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2657.290746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2657.324520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2658.584418] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2660.213532] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2660.314458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2661.839875] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2661.939403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2667.491151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2667.493552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2667.502818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2667.506274] Lustre: Skipped 1 previous similar message [ 2668.766832] Lustre: server umount lustre-OST0000 complete [ 2672.611108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.619486] Lustre: Skipped 1 previous similar message [ 2676.383987] Lustre: server umount lustre-MDT0000 complete [ 2677.727942] LustreError: 156682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.734669] LustreError: 156682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2677.776117] LustreError: 157447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442023 with bad export cookie 2231096596408507306 [ 2677.779360] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.781037] LustreError: 157447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2677.895633] Lustre: server umount lustre-MDT0001 complete [ 2680.910566] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:13:44 (1771442024) [ 2686.558643] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2689.922633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2704.671416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.791709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2704.813222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2704.855169] Lustre: lustre-MDT0000: new disk, initializing [ 2704.906355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.917143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2706.582533] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2711.764416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.812951] Lustre: 162983:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2711.949962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2713.673360] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2717.523321] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.224466] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.808215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.016990] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2729.489700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2729.552893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2729.915569] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2734.562460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2734.565277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.568235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.569304] Lustre: Skipped 2 previous similar messages [ 2734.571576] Lustre: Skipped 2 previous similar messages [ 2738.350859] Lustre: server umount lustre-OST0000 complete [ 2742.752787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2742.756266] LustreError: Skipped 1 previous similar message [ 2746.345515] Lustre: server umount lustre-MDT0000 complete [ 2748.195527] LustreError: 162975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442093 with bad export cookie 2231096596408508188 [ 2748.197868] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.200217] LustreError: 162975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2748.365140] Lustre: server umount lustre-MDT0001 complete [ 2752.825043] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:14:56 (1771442096) [ 2760.952087] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2765.474837] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2781.960549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.150296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2782.170175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2782.172720] Lustre: Skipped 3 previous similar messages [ 2782.228485] Lustre: lustre-MDT0000: new disk, initializing [ 2782.229930] Lustre: Skipped 2 previous similar messages [ 2782.269766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.273525] Lustre: Skipped 2 previous similar messages [ 2782.282430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2782.285411] Lustre: Skipped 2 previous similar messages [ 2784.315724] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2789.714883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.793163] Lustre: 168900:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2789.964242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2791.739986] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2795.178066] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2796.629553] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2799.427701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2801.712448] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2805.141303] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2805.323846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2810.340287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2810.341089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.341958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.341964] Lustre: Skipped 3 previous similar messages [ 2810.343956] LustreError: Skipped 1 previous similar message [ 2810.349205] Lustre: Skipped 3 previous similar messages [ 2813.846638] Lustre: server umount lustre-OST0000 complete [ 2821.359222] Lustre: server umount lustre-MDT0000 complete [ 2822.941342] LustreError: 168891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442168 with bad export cookie 2231096596408509259 [ 2822.943923] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.946219] LustreError: 168891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2823.076909] Lustre: server umount lustre-MDT0001 complete [ 2827.329307] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2828.605531] Key type lgssc unregistered [ 2828.749469] LNet: 172373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2828.751848] LNetError: 172373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2828.759326] LNet: Removed LNI 192.168.202.130@tcp [ 2829.149242] Key type .llcrypt unregistered [ 2829.150557] Key type ._llcrypt unregistered [ 2838.540221] Key type ._llcrypt registered [ 2838.541742] Key type .llcrypt registered [ 2838.600877] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2839.064402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.104610] alg: No test for adler32 (adler32-zlib) [ 2839.979694] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 2840.078786] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2841.679159] Key type lgssc registered [ 2842.112592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2845.178271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2845.194465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.420521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.248119] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2851.671731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.798632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2853.248312] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2855.215297] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2856.504715] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2859.168031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.338189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.754429] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2863.808119] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2870.637896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2888.160297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2888.165188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.171347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.233099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.234676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.234942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.242118] Lustre: Skipped 1 previous similar message [ 2892.313190] Lustre: server umount lustre-OST0000 complete [ 2896.352906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.353845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.359562] Lustre: Skipped 1 previous similar message [ 2901.472498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.478815] Lustre: Skipped 2 previous similar messages [ 2906.592242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.599047] Lustre: Skipped 1 previous similar message [ 2908.127207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2908.205463] Lustre: server umount lustre-MDT0000 complete [ 2910.169645] LustreError: 173728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442255 with bad export cookie 8554770708234383434 [ 2910.170344] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.173312] LustreError: 173728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.311963] Lustre: server umount lustre-MDT0001 complete [ 2914.687307] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2916.095534] Key type lgssc unregistered [ 2916.276547] LNet: 176956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2916.279507] LNetError: 176956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2916.288508] LNet: Removed LNI 192.168.202.130@tcp [ 2916.656640] Key type .llcrypt unregistered [ 2916.658108] Key type ._llcrypt unregistered [ 2927.337563] Key type ._llcrypt registered [ 2927.338827] Key type .llcrypt registered [ 2927.386771] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2927.829972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2927.909178] alg: No test for adler32 (adler32-zlib) [ 2928.785947] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 2928.881355] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2930.471230] Key type lgssc registered [ 2930.938343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2934.178457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2934.192673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.404549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.066604] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2940.727367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.929665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.436648] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2944.368230] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2945.802080] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2948.377287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2948.523054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.647608] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2952.501635] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2959.784940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2967.008619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2967.011563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.016890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.190658] Lustre: lustre-OST0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 2970.083129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2970.083310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.579601] Lustre: server umount lustre-OST0000 complete [ 2974.710764] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b1ae0750000.ost_server_uuid 50 [ 2975.344968] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2986.976135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2986.978758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.985108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2986.990140] Lustre: Skipped 2 previous similar messages [ 2991.072452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.073792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.837434] Lustre: server umount lustre-MDT0000 complete [ 2993.448579] LustreError: 178313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442338 with bad export cookie 6508395196384898702 [ 2993.451208] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2993.453317] LustreError: 178313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2993.599759] Lustre: server umount lustre-MDT0001 complete [ 2997.449436] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:19:01 (1771442341) [ 3000.335058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.562730] LustreError: 181814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.569309] LustreError: 181814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3000.588126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.282732] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3005.731920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.895385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3007.482783] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3009.451815] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.774771] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3013.253724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.757529] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3018.144302] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3021.033596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.149281] Lustre: lustre-OST0001: new disk, initializing [ 3021.151885] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3021.197649] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3021.200834] Lustre: Skipped 1 previous similar message [ 3023.820922] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3025.322943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3026.444706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3026.451685] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3026.499860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3027.445554] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3030.496422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3030.501308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.507715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3030.509763] Lustre: Skipped 1 previous similar message [ 3034.634415] Lustre: server umount lustre-OST0000 complete [ 3036.643230] LustreError: 183406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.649019] LustreError: 183406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3037.096805] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3037.194127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3037.789390] LustreError: 183406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.853692] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3038.955483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3041.761881] LustreError: 183405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.769048] LustreError: 183405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3046.883333] LustreError: 184832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.891288] LustreError: 184832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3052.000759] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3052.002864] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.004600] LustreError: Skipped 1 previous similar message [ 3052.009108] Lustre: Skipped 1 previous similar message [ 3052.011323] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3052.013942] Lustre: Skipped 4 previous similar messages [ 3055.410121] Lustre: server umount lustre-OST0001 complete [ 3063.008151] Lustre: server umount lustre-MDT0000 complete [ 3064.609245] LustreError: 181796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442410 with bad export cookie 6508395196384899759 [ 3064.612160] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.613080] LustreError: 181796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3064.761507] Lustre: server umount lustre-MDT0001 complete [ 3074.762985] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:20:18 (1771442418) [ 3077.455050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.603235] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3077.619702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3077.702475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.187424] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3082.464463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.503976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3082.523959] Lustre: 187588:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3084.157995] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3086.249794] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3087.594740] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3090.175738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.286803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3090.904212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3092.719134] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3094.815889] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3097.574429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.614646] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3099.949976] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3101.977521] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3103.200941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.207490] Lustre: Skipped 3 previous similar messages [ 3103.208485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3103.212351] Lustre: Skipped 5 previous similar messages [ 3108.989721] Lustre: server umount lustre-OST0000 complete [ 3110.887429] LustreError: 189315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.894308] LustreError: 189315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3121.121926] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3121.125166] LustreError: Skipped 2 previous similar messages [ 3125.911164] Lustre: server umount lustre-OST0001 complete [ 3133.539989] Lustre: server umount lustre-MDT0000 complete [ 3135.114384] LustreError: 187578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442480 with bad export cookie 6508395196384901145 [ 3135.115088] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3135.118701] LustreError: 187578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3135.256801] Lustre: server umount lustre-MDT0001 complete [ 3144.870618] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:21:28 (1771442488) [ 3150.933479] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 3154.308351] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3167.578386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.697838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3167.714482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3167.762043] Lustre: lustre-MDT0000: new disk, initializing [ 3167.791342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.794052] Lustre: Skipped 3 previous similar messages [ 3167.801271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3169.247121] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3174.051118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.100187] Lustre: 195160:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3174.125416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3174.127858] Lustre: Skipped 1 previous similar message [ 3174.171497] Lustre: lustre-MDT0001: new disk, initializing [ 3174.219684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3174.226200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3175.724633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3178.960532] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3180.289144] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3182.596614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.709823] Lustre: lustre-OST0000: new disk, initializing [ 3182.712343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3184.671483] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3187.197774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3187.201222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3187.244873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3187.894557] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3189.330836] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3189.410717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3190.811348] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3190.883689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3192.289955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.290860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.293966] LustreError: Skipped 2 previous similar messages [ 3192.297653] Lustre: Skipped 4 previous similar messages [ 3192.298285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3192.302948] Lustre: Skipped 9 previous similar messages [ 3193.696073] Lustre: server umount lustre-OST0000 complete [ 3201.317890] Lustre: server umount lustre-MDT0000 complete [ 3202.528277] LustreError: 195172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.534408] LustreError: 195172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3202.735414] LustreError: 195153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442548 with bad export cookie 6508395196384902664 [ 3202.738903] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.740047] LustreError: 195153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3202.873422] Lustre: server umount lustre-MDT0001 complete [ 3206.328442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.995395] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3211.031996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.451800] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3214.381424] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3215.681244] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3218.150866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.310086] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3222.090100] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3223.470624] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3230.709314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3232.078290] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3232.162925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3235.809790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3238.932599] Lustre: server umount lustre-OST0000 complete [ 3241.009337] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3242.430816] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3265.968915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.110770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.114979] Lustre: Skipped 5 previous similar messages [ 3268.049767] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3269.772026] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3270.296967] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3276.320403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3276.322009] Lustre: Skipped 5 previous similar messages [ 3277.333108] Lustre: server umount lustre-OST0000 complete [ 3281.376995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.381055] Lustre: Skipped 6 previous similar messages [ 3284.629636] Lustre: server umount lustre-MDT0000 complete [ 3286.001253] LustreError: 203305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442631 with bad export cookie 6508395196384903532 [ 3286.003878] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.004976] LustreError: 203305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3286.131282] Lustre: server umount lustre-MDT0001 complete [ 3288.676628] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:23:52 (1771442632) [ 3290.808583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.980221] LustreError: 204466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.987467] LustreError: 204466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3292.355984] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3295.243885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.573018] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3298.228200] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3299.343623] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3301.391261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.239467] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3304.776151] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3306.204620] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3306.278864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3307.701896] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3307.790284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3309.906258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.977260] Lustre: lustre-OST0001: new disk, initializing [ 3309.979174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3311.745020] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3313.277841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3313.282090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3313.311887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3314.658873] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3315.848831] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3315.916595] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3317.053395] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3317.118883] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3318.756868] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3319.745546] Lustre: server umount lustre-OST0001 complete [ 3321.740501] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3321.818185] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3323.187844] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3323.260799] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3346.860431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.933343] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3350.626528] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3351.170968] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3357.880450] Lustre: server umount lustre-OST0001 complete [ 3362.275042] LustreError: 206252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.279505] LustreError: 206252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 3384.287102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3384.342536] Lustre: server umount lustre-OST0000 complete [ 3391.768538] Lustre: server umount lustre-MDT0000 complete [ 3392.898197] LustreError: 204448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442738 with bad export cookie 6508395196384904463 [ 3392.900809] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.901906] LustreError: 204448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3407.327119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3407.412585] Lustre: server umount lustre-MDT0001 complete [ 3414.841662] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:25:58 (1771442758) [ 3416.872735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.963595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3416.972832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3417.042530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.044590] Lustre: Skipped 5 previous similar messages [ 3418.213717] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3420.884386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.917133] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3420.925945] Lustre: 212411:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3422.184721] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3423.998263] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3425.265976] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3427.587299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3427.676832] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3429.636693] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3431.358037] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3440.054599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3440.082681] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3441.816505] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3443.404864] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3446.938246] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3448.113065] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3448.176087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3449.334552] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3449.396282] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3450.484516] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b1ac3ef1000.ost_server_uuid 50 [ 3450.927078] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b1ac3ef1000.ost_server_uuid in FULL state after 0 sec [ 3451.228763] Lustre: Permanently deactivating lustre-OST0001 [ 3451.234927] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3451.963658] Lustre: Permanently reactivating lustre-OST0001 [ 3456.993659] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.995839] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.995942] LustreError: Skipped 4 previous similar messages [ 3456.999033] Lustre: Skipped 6 previous similar messages [ 3456.999529] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3457.003052] Lustre: Skipped 21 previous similar messages [ 3458.937636] Lustre: server umount lustre-OST0001 complete [ 3474.399124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3474.456568] Lustre: server umount lustre-OST0000 complete [ 3490.271100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3490.323302] Lustre: server umount lustre-MDT0000 complete [ 3491.491215] LustreError: 216229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442836 with bad export cookie 6508395196384905723 [ 3491.494468] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3491.495912] LustreError: 216229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3491.607808] Lustre: server umount lustre-MDT0001 complete [ 3498.829182] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:27:22 (1771442842) [ 3501.415661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.515770] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3501.528678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3501.532652] Lustre: Skipped 5 previous similar messages [ 3502.869185] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3505.493926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.525538] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3505.534979] Lustre: 218509:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.835790] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3508.461255] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3509.568974] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3511.620733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3511.692153] Lustre: Permanently deactivating lustre-OST0000 [ 3511.695915] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3513.450126] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3515.241660] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3515.248757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3517.472672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.532364] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3521.244507] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3524.623942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3527.166904] Lustre: Permanently reactivating lustre-OST0000 [ 3532.273364] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3532.301720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3537.088969] Lustre: server umount lustre-OST0001 complete [ 3539.939333] LustreError: 220129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.944743] LustreError: 220129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 3544.468110] Lustre: server umount lustre-OST0000 complete [ 3552.031252] Lustre: server umount lustre-MDT0000 complete [ 3553.204115] LustreError: 219235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442898 with bad export cookie 6508395196384907361 [ 3553.207157] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.207384] LustreError: 219235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3553.316939] Lustre: server umount lustre-MDT0001 complete [ 3556.619434] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3557.554996] Key type lgssc unregistered [ 3557.676548] LNet: 222890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3557.678693] LNetError: 222890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3557.686325] LNet: Removed LNI 192.168.202.130@tcp [ 3557.960787] Key type .llcrypt unregistered [ 3557.962338] Key type ._llcrypt unregistered [ 3565.956684] Key type ._llcrypt registered [ 3565.957908] Key type .llcrypt registered [ 3565.990761] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3566.361596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3566.390251] alg: No test for adler32 (adler32-zlib) [ 3567.235782] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 3567.308952] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3568.887120] Key type lgssc registered [ 3569.198415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3572.154284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3572.159425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.299114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.415345] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3576.929032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.966122] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3576.975641] Lustre: 224305:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3576.980231] Lustre: Permanently deactivating lustre-MDT0001 [ 3576.984794] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3577.081953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3578.275074] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3579.966381] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3581.111123] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3581.930131] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3581.932721] LustreError: 224347:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3581.935964] LustreError: 224347:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3581.938962] LustreError: 224347:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3583.213071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.334848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.007591] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3586.610785] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3587.427157] LustreError: 225930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.434242] LustreError: 225930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3588.706611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.779425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3590.538724] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3591.475781] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3592.218136] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3597.606794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3597.607369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3603.081072] Lustre: Permanently reactivating lustre-MDT0001 [ 3603.086650] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3603.088710] Lustre: Skipped 2 previous similar messages [ 3606.241761] Lustre: lustre-MDT0001: Client e9fa3715-0d61-4d11-8ad6-3ab752c5d6e5 (at 192.168.202.30@tcp) reconnecting [ 3608.220168] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3608.691825] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3610.210487] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3610.277182] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3611.495955] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3611.564596] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3612.313115] Lustre: Permanently deactivating lustre-MDT0001 [ 3612.317716] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3612.320434] Lustre: Skipped 2 previous similar messages [ 3618.814985] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3621.030875] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3621.035044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.042150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.392825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.394713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.399219] Lustre: Skipped 2 previous similar messages [ 3627.297828] Lustre: server umount lustre-MDT0000 complete [ 3628.502899] LustreError: 224296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771442973 with bad export cookie 17637685497461429586 [ 3628.506277] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3628.506682] LustreError: 224296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3628.511573] LustreError: 225789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3628.517690] LustreError: 225789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3628.618468] Lustre: server umount lustre-MDT0001 complete [ 3635.836882] Lustre: server umount lustre-OST0000 complete [ 3639.335358] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:29:43 (1771442983) [ 3642.600459] Lustre: server umount lustre-OST0001 complete [ 3644.534478] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 3647.522881] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3658.451444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.529708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3658.532743] Lustre: Skipped 2 previous similar messages [ 3658.541209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3658.577927] Lustre: lustre-MDT0000: new disk, initializing [ 3658.603140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.609488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3659.713951] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3663.555480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.595646] Lustre: 232020:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3663.616067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3663.617638] Lustre: Skipped 1 previous similar message [ 3663.653855] Lustre: lustre-MDT0001: new disk, initializing [ 3663.696067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3663.699560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3664.996891] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3668.568708] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3670.434560] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3673.690634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.853812] Lustre: lustre-OST0000: new disk, initializing [ 3673.857961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3673.914162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.918161] Lustre: Skipped 1 previous similar message [ 3674.511488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3674.519694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3674.566430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3676.672639] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3680.983326] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3682.129964] LustreError: 232027:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3687.198025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3687.270519] Lustre: lustre-OST0001: new disk, initializing [ 3687.272945] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3689.380063] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3692.223082] LustreError: 232027:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3692.232474] LustreError: 232027:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3692.609370] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3693.204799] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3694.591475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3694.594668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3694.627505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3702.327118] LustreError: 232027:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3704.805404] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.808428] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.810651] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3704.812349] Lustre: Skipped 1 previous similar message [ 3704.814175] Lustre: Skipped 3 previous similar messages [ 3708.006443] Lustre: lustre-OST0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 3708.008643] Lustre: Skipped 1 previous similar message [ 3709.061170] Lustre: server umount lustre-OST0001 complete [ 3709.924221] LustreError: 235324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.930297] LustreError: 235324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3715.045510] LustreError: 235324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.049527] LustreError: 235324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3720.163692] LustreError: 233746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.170853] LustreError: 233746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3725.283420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.284246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.288510] Lustre: Skipped 1 previous similar message [ 3725.291840] Lustre: Skipped 1 previous similar message [ 3727.176644] Lustre: server umount lustre-OST0000 complete [ 3730.400832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3730.401267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.403665] LustreError: Skipped 1 previous similar message [ 3734.491655] Lustre: server umount lustre-MDT0000 complete [ 3735.519842] LustreError: 232962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.524332] LustreError: 232962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3735.709151] LustreError: 232796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443081 with bad export cookie 17637685497461432155 [ 3735.712435] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3735.713015] LustreError: 232796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3749.855138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3749.954564] Lustre: server umount lustre-MDT0001 complete [ 3753.194659] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3754.123031] Key type lgssc unregistered [ 3754.240342] LNet: 236666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3754.243233] LNetError: 236666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3754.256256] LNet: Removed LNI 192.168.202.130@tcp [ 3754.541384] Key type .llcrypt unregistered [ 3754.542976] Key type ._llcrypt unregistered [ 3768.016966] Key type ._llcrypt registered [ 3768.018630] Key type .llcrypt registered [ 3768.060721] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3768.483504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3768.514482] alg: No test for adler32 (adler32-zlib) [ 3769.372143] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 3769.463236] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3771.047166] Key type lgssc registered [ 3771.437082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3774.085385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3774.091979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3775.194753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3775.202070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3775.257809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.446668] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3779.128827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.163171] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3779.174278] Lustre: 238832:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3779.261735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3780.423314] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3782.053185] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3783.146528] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3785.304718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3785.377270] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3785.439706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3787.204646] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3788.874849] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3791.882031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3832.801554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.801761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.806394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.810860] Lustre: Skipped 1 previous similar message [ 3837.922074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.923793] Lustre: Skipped 1 previous similar message [ 3838.097296] Lustre: server umount lustre-OST0000 complete [ 3839.454383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3843.398318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.517039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.301063] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3847.020824] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3863.015069] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3863.018977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3863.019265] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3863.024207] Lustre: Skipped 1 previous similar message [ 3883.492508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.493660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3883.494374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3883.497256] Lustre: Skipped 2 previous similar messages [ 3883.499867] LustreError: Skipped 1 previous similar message [ 3886.295270] Lustre: server umount lustre-OST0000 complete [ 3888.608502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.609390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3888.611933] Lustre: Skipped 1 previous similar message [ 3888.613850] Lustre: Skipped 1 previous similar message [ 3893.663400] Lustre: server umount lustre-MDT0000 complete [ 3894.948051] LustreError: 241085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443240 with bad export cookie 13530996577002557534 [ 3894.950843] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3894.951947] LustreError: 241085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3895.065826] Lustre: server umount lustre-MDT0001 complete [ 3898.434509] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3899.428236] Key type lgssc unregistered [ 3899.561416] LNet: 243779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3899.563745] LNetError: 243779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3899.574321] LNet: Removed LNI 192.168.202.130@tcp [ 3899.877203] Key type .llcrypt unregistered [ 3899.878496] Key type ._llcrypt unregistered [ 3908.816223] Key type ._llcrypt registered [ 3908.817222] Key type .llcrypt registered [ 3908.855689] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3909.216426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3909.282187] alg: No test for adler32 (adler32-zlib) [ 3910.128510] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 3910.212404] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3911.791118] Key type lgssc registered [ 3912.166128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.670693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.676305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.829929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3917.053919] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3919.786711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.912181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3921.144207] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3922.813209] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3924.118619] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3926.297461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3926.412796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.205208] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3929.877274] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3940.773267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3956.194513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.194586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3956.195694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.197839] Lustre: Skipped 1 previous similar message [ 3956.203224] LustreError: Skipped 1 previous similar message [ 3959.184291] Lustre: server umount lustre-OST0000 complete [ 3960.800119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3960.802807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.807185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.810409] Lustre: Skipped 1 previous similar message [ 3966.432769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3966.434634] Lustre: Skipped 3 previous similar messages [ 3966.621081] Lustre: server umount lustre-MDT0000 complete [ 3967.803157] LustreError: 248392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443313 with bad export cookie 12706354883378340962 [ 3967.806440] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3967.806531] LustreError: 248392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3967.931593] Lustre: server umount lustre-MDT0001 complete [ 3971.588588] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3972.730935] Key type lgssc unregistered [ 3972.880408] LNet: 249125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3972.883329] LNetError: 249125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3972.891559] LNet: Removed LNI 192.168.202.130@tcp [ 3973.221077] Key type .llcrypt unregistered [ 3973.222690] Key type ._llcrypt unregistered [ 3980.329586] Key type ._llcrypt registered [ 3980.330705] Key type .llcrypt registered [ 3980.373892] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3980.808673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3980.850962] alg: No test for adler32 (adler32-zlib) [ 3981.715930] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 3981.806893] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3983.391208] Key type lgssc registered [ 3983.807780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3986.519272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3986.525280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.677944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.056692] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3992.159882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.294256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3993.668024] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3995.516870] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3996.707942] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3998.883061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3998.990624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.814809] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4002.645913] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4014.375220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4022.751895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4022.755495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.761092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.801449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4024.802922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.804292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.804608] LustreError: Skipped 1 previous similar message [ 4024.806759] Lustre: Skipped 1 previous similar message [ 4027.848835] Lustre: server umount lustre-OST0000 complete [ 4029.920320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.924680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.926771] Lustre: Skipped 1 previous similar message [ 4031.127407] Lustre: server umount lustre-MDT0000 complete [ 4032.365206] LustreError: 250290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443377 with bad export cookie 8673053630373503023 [ 4032.368696] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4032.370766] LustreError: 250290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4038.510093] Lustre: server umount lustre-MDT0001 complete [ 4041.983240] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4042.998659] Key type lgssc unregistered [ 4043.131365] LNet: 253653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4043.135731] LNetError: 253653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4043.147304] LNet: Removed LNI 192.168.202.130@tcp [ 4043.441146] Key type .llcrypt unregistered [ 4043.442275] Key type ._llcrypt unregistered [ 4052.177219] Key type ._llcrypt registered [ 4052.178697] Key type .llcrypt registered [ 4052.220179] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4052.678208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4052.685106] alg: No test for adler32 (adler32-zlib) [ 4053.538911] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4053.621485] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4055.207116] Key type lgssc registered [ 4055.558175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4058.017378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4058.022602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.172424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.380835] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4063.080905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.196230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4064.407311] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4066.119640] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4067.292574] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4069.467631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4069.575832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.356971] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4073.041949] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4083.940706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4096.992827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4096.994886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.998158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4098.528154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.533517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.540948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.553140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.562952] Lustre: Skipped 1 previous similar message [ 4102.254788] Lustre: server umount lustre-OST0000 complete [ 4104.673986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4104.680178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.280780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.284695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.292280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.495826] Lustre: server umount lustre-MDT0000 complete [ 4112.326129] LustreError: 255011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443457 with bad export cookie 10061436392067975264 [ 4112.327728] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.334873] LustreError: 255011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4112.494942] Lustre: server umount lustre-MDT0001 complete [ 4117.542450] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4118.939333] Key type lgssc unregistered [ 4119.108359] LNet: 259001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4119.113311] LNetError: 259001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4119.122834] LNet: Removed LNI 192.168.202.130@tcp [ 4119.554790] Key type .llcrypt unregistered [ 4119.557059] Key type ._llcrypt unregistered [ 4128.796810] Key type ._llcrypt registered [ 4128.798540] Key type .llcrypt registered [ 4128.863634] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4129.404412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4129.411984] alg: No test for adler32 (adler32-zlib) [ 4130.353302] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4130.476302] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4132.095221] Key type lgssc registered [ 4132.673307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4136.183923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4136.191326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.393869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.761831] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4141.855131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.987143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.367058] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4145.296508] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4146.599899] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4149.043048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4149.160230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.103059] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4152.814225] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4162.470396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4167.648760] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4167.651518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.656229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4167.658634] Lustre: Skipped 1 previous similar message [ 4168.920893] Lustre: server umount lustre-OST0000 complete [ 4172.313136] Lustre: server umount lustre-MDT0000 complete [ 4172.768358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.768494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4172.769181] LustreError: 260192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4172.769189] LustreError: 260192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4172.773045] Lustre: Skipped 2 previous similar messages [ 4172.776834] LustreError: Skipped 1 previous similar message [ 4173.574644] LustreError: 260164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443519 with bad export cookie 14975547203525376331 [ 4173.578187] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4173.580054] LustreError: 260164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4173.692756] Lustre: server umount lustre-MDT0001 complete [ 4177.459934] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4178.563901] Key type lgssc unregistered [ 4178.687400] LNet: 263561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4178.690328] LNetError: 263561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4178.698558] LNet: Removed LNI 192.168.202.130@tcp [ 4179.043778] Key type .llcrypt unregistered [ 4179.046079] Key type ._llcrypt unregistered [ 4182.933990] Key type ._llcrypt registered [ 4182.935418] Key type .llcrypt registered [ 4182.977345] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4188.719385] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4194.945089] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4195.399884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4195.417107] alg: No test for adler32 (adler32-zlib) [ 4196.276252] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4196.361971] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4197.943161] Key type lgssc registered [ 4198.306596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4209.109867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4209.114269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.194986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4210.206768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4210.239670] Lustre: lustre-MDT0000: new disk, initializing [ 4210.260674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.267705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4211.504409] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4215.805677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.851056] Lustre: 267034:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4215.877296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4215.880255] Lustre: Skipped 1 previous similar message [ 4215.918077] Lustre: lustre-MDT0001: new disk, initializing [ 4215.957861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.968823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4215.971469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4217.289303] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4220.434337] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4221.601250] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4223.756697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.855064] Lustre: lustre-OST0000: new disk, initializing [ 4223.856849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4223.881858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.648749] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4228.761427] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4230.135319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4230.138870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4230.179569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4230.241779] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4230.319529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4231.785167] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4231.860943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4235.235624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.237596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.242932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.244744] LustreError: Skipped 1 previous similar message [ 4235.247299] Lustre: Skipped 1 previous similar message [ 4238.755501] Lustre: server umount lustre-OST0000 complete [ 4240.352317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.352919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4240.358088] Lustre: Skipped 2 previous similar messages [ 4245.472066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4245.473955] Lustre: Skipped 2 previous similar messages [ 4246.367136] Lustre: server umount lustre-MDT0000 complete [ 4247.725191] LustreError: 267027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443593 with bad export cookie 11746798155087501731 [ 4247.728700] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.730200] LustreError: 267027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4247.847090] Lustre: server umount lustre-MDT0001 complete [ 4250.692360] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:39:54 (1771443594) [ 4252.962422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.159938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.587326] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4257.644542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.789303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4259.258828] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4261.199160] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4262.498753] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4264.988882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4267.315683] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4269.346531] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.899303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4284.899805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.902262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4284.905920] Lustre: Skipped 1 previous similar message [ 4290.012722] Lustre: server umount lustre-OST0000 complete [ 4293.600256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4293.603551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.609771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.612499] Lustre: Skipped 1 previous similar message [ 4297.835244] Lustre: server umount lustre-MDT0000 complete [ 4299.245325] LustreError: 271019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443644 with bad export cookie 11746798155087502613 [ 4299.248789] LustreError: 271757:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4299.249344] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4299.249915] LustreError: 271019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4299.376173] Lustre: server umount lustre-MDT0001 complete [ 4303.721139] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4304.932522] Key type lgssc unregistered [ 4305.094567] LNet: 274241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4305.098718] LNetError: 274241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4305.109422] LNet: Removed LNI 192.168.202.130@tcp [ 4305.521443] Key type .llcrypt unregistered [ 4305.522439] Key type ._llcrypt unregistered [ 4315.512425] Key type ._llcrypt registered [ 4315.513516] Key type .llcrypt registered [ 4315.563431] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4321.899438] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4322.377595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4322.386119] alg: No test for adler32 (adler32-zlib) [ 4323.244574] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4323.335755] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4324.919194] Key type lgssc registered [ 4325.305250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4341.316305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4341.321338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.425938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4342.444321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4342.488661] Lustre: lustre-MDT0000: new disk, initializing [ 4342.525773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4342.535844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4343.944437] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4348.654675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.706146] Lustre: 278005:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4348.732422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4348.735292] Lustre: Skipped 1 previous similar message [ 4348.772471] Lustre: lustre-MDT0001: new disk, initializing [ 4348.807134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4348.825359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4348.829926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4350.549636] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4353.963435] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4355.424723] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4358.093904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4358.210612] Lustre: lustre-OST2710: new disk, initializing [ 4358.213835] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4358.244701] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4359.678274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4359.682992] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4359.732583] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4360.437235] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4363.897634] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4366.553961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4366.646483] Lustre: lustre-OST03e8: new disk, initializing [ 4366.652033] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4366.698988] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4368.886625] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4371.199821] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4372.480458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4372.487548] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4372.524874] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4375.360510] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4375.501430] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4377.292811] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4377.389657] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4379.336452] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4379.441348] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4381.224395] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4381.326870] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4382.688375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.692127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4382.695076] Lustre: Skipped 2 previous similar messages [ 4382.696944] Lustre: Skipped 1 previous similar message [ 4387.809487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4387.812080] Lustre: Skipped 5 previous similar messages [ 4388.716767] Lustre: server umount lustre-MDT0000 complete [ 4390.359133] LustreError: 280697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443735 with bad export cookie 5586317831904241109 [ 4390.364284] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4390.368289] LustreError: 280697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4390.515712] Lustre: server umount lustre-MDT0001 complete [ 4402.202666] Lustre: server umount lustre-OST2710 complete [ 4414.259351] Lustre: server umount lustre-OST03e8 complete [ 4422.682847] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4426.379721] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4439.283284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.400601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4439.413267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4439.452880] Lustre: lustre-MDT0000: new disk, initializing [ 4439.485074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.494034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4440.939124] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4445.322692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.368333] Lustre: 285723:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4445.493656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4446.855409] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4449.966777] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4451.280517] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4453.588911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.686168] Lustre: lustre-OST0000: new disk, initializing [ 4453.687923] Lustre: Skipped 1 previous similar message [ 4453.689520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4453.691537] Lustre: Skipped 2 previous similar messages [ 4453.719469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.722161] Lustre: Skipped 1 previous similar message [ 4455.568311] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4458.674829] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4460.095960] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4461.056283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4461.059458] Lustre: Skipped 1 previous similar message [ 4461.061755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4461.097489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4461.189484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4462.574544] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4462.654957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4466.146691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4466.147027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.149914] LustreError: Skipped 1 previous similar message [ 4466.155075] Lustre: Skipped 1 previous similar message [ 4466.156041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4466.161125] Lustre: Skipped 1 previous similar message [ 4469.520922] Lustre: server umount lustre-OST0000 complete [ 4471.265036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4471.265385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.270800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4471.276160] Lustre: Skipped 2 previous similar messages [ 4476.384179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.386210] Lustre: Skipped 2 previous similar messages [ 4477.087341] Lustre: server umount lustre-MDT0000 complete [ 4478.449966] LustreError: 287471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443823 with bad export cookie 5586317831904242775 [ 4478.450652] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.456623] LustreError: 287471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4478.582055] Lustre: server umount lustre-MDT0001 complete [ 4481.587885] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:43:45 (1771443825) [ 4482.113628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4482.682808] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:43:46 (1771443826) [ 4483.730255] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4493.312211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.416041] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4493.423662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4493.486844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.895814] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4497.779191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.811702] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4497.822519] Lustre: 291156:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4499.147209] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4500.905270] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4502.095124] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4515.368591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4515.463468] Lustre: Denying initial registration attempt from nid 192.168.202.130@tcp, specified as failover [ 4515.467678] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4515.471529] LustreError: 292768:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 4515.475314] LustreError: 292768:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4515.478200] LustreError: 292768:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4515.507384] Lustre: server umount lustre-OST0000 complete [ 4515.510040] LustreError: 292768:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4518.368593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4518.373970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.380196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.466032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.113924] Lustre: server umount lustre-MDT0000 complete [ 4525.371342] LustreError: 291147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443870 with bad export cookie 5586317831904243643 [ 4525.374198] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.375115] LustreError: 291147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4525.481533] Lustre: server umount lustre-MDT0001 complete [ 4529.100106] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4530.196749] Key type lgssc unregistered [ 4530.340366] LNet: 293868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4530.342872] LNetError: 293868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4530.353656] LNet: Removed LNI 192.168.202.130@tcp [ 4530.669515] Key type .llcrypt unregistered [ 4530.670576] Key type ._llcrypt unregistered [ 4534.477261] Key type ._llcrypt registered [ 4534.478461] Key type .llcrypt registered [ 4534.524817] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4534.902970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4534.944929] alg: No test for adler32 (adler32-zlib) [ 4535.806594] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4535.898700] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4537.487173] Key type lgssc registered [ 4537.874188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4573.217922] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4576.239513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4576.247060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.339849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4577.352318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4577.408434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.809162] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4581.920475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.965974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4581.976876] Lustre: 296580:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4582.072238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4583.451863] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4585.229749] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4586.425979] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4599.713555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.797196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4599.857852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.926274] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4603.835326] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4604.649263] Lustre: server umount lustre-OST0000 complete [ 4607.968460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4607.971334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.976372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.503331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.503980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.253116] Lustre: server umount lustre-MDT0000 complete [ 4613.470329] LustreError: 297303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771443958 with bad export cookie 16501615940995083177 [ 4613.472514] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4613.474843] LustreError: 297303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4613.596593] Lustre: server umount lustre-MDT0001 complete [ 4617.141147] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4618.213791] Key type lgssc unregistered [ 4618.338499] LNet: 299786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4618.341479] LNetError: 299786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4618.349304] LNet: Removed LNI 192.168.202.130@tcp [ 4618.640437] Key type .llcrypt unregistered [ 4618.642214] Key type ._llcrypt unregistered [ 4628.529636] Key type ._llcrypt registered [ 4628.530946] Key type .llcrypt registered [ 4628.571476] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4629.010723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4629.046981] alg: No test for adler32 (adler32-zlib) [ 4629.939412] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4630.042529] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4631.631161] Key type lgssc registered [ 4632.013538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4634.826617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4634.836254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.015578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4637.390202] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4640.678219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.843530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4642.262583] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4644.244517] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4645.492574] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4647.889893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.012258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.979785] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4651.898440] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4671.455857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4671.459107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.465085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4672.615702] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 4672.995490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.068034] Lustre: server umount lustre-MDT0000 complete [ 4677.621409] LustreError: 302779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444023 with bad export cookie 17127652105171801757 [ 4677.622321] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4677.626048] LustreError: 302779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4677.725043] LustreError: 301160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.731771] LustreError: 301160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4677.733423] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 4677.738894] Lustre: Skipped 3 previous similar messages [ 4678.112515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.120321] Lustre: Skipped 1 previous similar message [ 4682.143676] LustreError: 301158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.143940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4682.150035] LustreError: 301158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4682.152091] Lustre: Skipped 1 previous similar message [ 4683.838920] Lustre: server umount lustre-MDT0001 complete [ 4686.447852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4689.657468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.623698] LustreError: 300472:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091762e8d x1857493637556352/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4702.740761] LustreError: 302768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.747845] LustreError: 302768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4702.770976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.792094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4704.219354] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4706.848828] LustreError: 304438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.856318] LustreError: 304438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4707.429330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.559361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4709.001681] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4711.254867] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4711.652782] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4714.039547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4715.541311] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4716.245762] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4718.048806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4718.049943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.054870] LustreError: Skipped 1 previous similar message [ 4718.060449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4718.070403] Lustre: Skipped 2 previous similar messages [ 4727.265213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4727.267801] Lustre: Skipped 3 previous similar messages [ 4732.383974] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4732.450755] Lustre: server umount lustre-OST0000 complete [ 4737.504909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4737.509807] Lustre: Skipped 2 previous similar messages [ 4747.745078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4747.747977] Lustre: Skipped 7 previous similar messages [ 4748.255098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4748.325300] Lustre: server umount lustre-MDT0000 complete [ 4749.738586] LustreError: 306137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444095 with bad export cookie 17127652105171808911 [ 4749.741655] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4749.742634] LustreError: 306137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4749.872981] Lustre: server umount lustre-MDT0001 complete [ 4753.864028] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4754.961515] Key type lgssc unregistered [ 4755.103356] LNet: 306871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4755.107707] LNetError: 306871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4755.115323] LNet: Removed LNI 192.168.202.130@tcp [ 4755.450749] Key type .llcrypt unregistered [ 4755.452036] Key type ._llcrypt unregistered [ 4765.452290] Key type ._llcrypt registered [ 4765.453646] Key type .llcrypt registered [ 4765.503993] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4765.988304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4766.003056] alg: No test for adler32 (adler32-zlib) [ 4766.869341] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4766.964119] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4768.543168] Key type lgssc registered [ 4768.978776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4771.779438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4771.786494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.976892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.494943] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4777.575950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.716549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4779.148421] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4781.127106] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4782.357895] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4788.192652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4788.192830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.203301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.208164] Lustre: Skipped 1 previous similar message [ 4789.217530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4789.219620] Lustre: Skipped 1 previous similar message [ 4789.526837] Lustre: server umount lustre-MDT0000 complete [ 4790.791534] LustreError: 308244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444136 with bad export cookie 12022248718524355205 [ 4790.791942] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4790.795674] LustreError: 308244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4790.896923] Lustre: server umount lustre-MDT0001 complete [ 4795.983450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.088679] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4796.101031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4796.179345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.529818] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4800.476382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.512057] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4800.523378] Lustre: 310620:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4800.628174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4801.982227] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4803.785206] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4805.017834] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4807.588689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4807.657525] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4807.661618] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4807.665460] LustreError: 312276:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 4807.668991] LustreError: 312276:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 4807.671640] LustreError: 312276:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4807.691486] Lustre: server umount lustre-OST0000 complete [ 4807.693170] LustreError: 312276:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4810.544343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4810.609379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4810.669585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4812.562125] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4814.485549] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4817.317441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4817.353181] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4817.377647] Lustre: lustre-OST0001: new disk, initializing [ 4817.379556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4819.425928] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4820.301067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4820.307338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4820.324914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4820.358213] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4821.335990] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4825.573080] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.573624] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4825.580092] Lustre: Skipped 1 previous similar message [ 4825.586404] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4825.588877] Lustre: Skipped 1 previous similar message [ 4828.482823] Lustre: server umount lustre-OST0001 complete [ 4830.689404] LustreError: 312740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.690679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4830.691724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.691729] Lustre: Skipped 1 previous similar message [ 4830.692213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4830.696235] LustreError: 312740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4830.698817] LustreError: Skipped 2 previous similar messages [ 4835.809909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4835.810773] LustreError: 312739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.813056] Lustre: Skipped 1 previous similar message [ 4835.920097] Lustre: server umount lustre-OST0000 complete [ 4840.929997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.937207] Lustre: Skipped 2 previous similar messages [ 4843.435667] Lustre: server umount lustre-MDT0000 complete [ 4844.956415] LustreError: 311343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444190 with bad export cookie 12022248718524355744 [ 4844.962508] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4845.211056] Lustre: server umount lustre-MDT0001 complete [ 4849.099364] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4850.256284] Key type lgssc unregistered [ 4850.423877] LNet: 315520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4850.428284] LNetError: 315520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4850.441126] LNet: Removed LNI 192.168.202.130@tcp [ 4850.769185] Key type .llcrypt unregistered [ 4850.770456] Key type ._llcrypt unregistered [ 4883.844100] Key type ._llcrypt registered [ 4883.845303] Key type .llcrypt registered [ 4883.883753] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4889.937105] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4890.401163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4890.416206] alg: No test for adler32 (adler32-zlib) [ 4891.279042] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 4891.368594] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4892.951133] Key type lgssc registered [ 4893.332021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4904.711490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4904.719298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4905.811407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4905.825183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4905.862526] Lustre: lustre-MDT0000: new disk, initializing [ 4905.893163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4905.903838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4907.318287] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4911.854906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.903469] Lustre: 320392:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4911.931736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4911.934269] Lustre: Skipped 1 previous similar message [ 4911.969373] Lustre: lustre-MDT0001: new disk, initializing [ 4912.007870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4912.023051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4912.027168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4913.353897] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4916.574125] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.819160] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4920.148802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4920.279150] Lustre: lustre-OST0000: new disk, initializing [ 4920.281596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4920.319575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4921.282045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4921.286495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4921.334013] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4922.352842] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4925.684694] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4927.260272] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4927.342191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4928.866752] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4928.951785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4931.555175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4931.555631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.557435] LustreError: Skipped 1 previous similar message [ 4931.561368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4931.565039] Lustre: Skipped 1 previous similar message [ 4935.692586] Lustre: server umount lustre-OST0000 complete [ 4937.696215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4937.699703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.703949] Lustre: Skipped 1 previous similar message [ 4937.706369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.792156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.792546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.065374] Lustre: server umount lustre-MDT0000 complete [ 4944.330146] LustreError: 321169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444289 with bad export cookie 6305784624038747022 [ 4944.332776] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.333378] LustreError: 321169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4944.449160] Lustre: server umount lustre-MDT0001 complete [ 4947.171826] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 14:51:31 (1771444291) [ 4949.508784] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 14:51:33 (1771444293) [ 4952.275936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.435859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.632993] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4956.281258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.696050] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4959.489248] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4960.614382] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4962.720051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.785741] LustreError: 326329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.790266] LustreError: 326329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4962.815996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4962.818301] Lustre: Skipped 1 previous similar message [ 4964.545561] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4966.288073] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4970.608222] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4971.187051] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4971.746474] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4972.292176] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4972.920812] Lustre: Failing over lustre-MDT0000 [ 4973.076837] Lustre: server umount lustre-MDT0000 complete [ 4976.609135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.610547] LustreError: 324754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.613257] Lustre: Skipped 2 previous similar messages [ 4976.619673] LustreError: 324754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4977.762483] LustreError: 326147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.728298] LustreError: 327035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.733065] LustreError: 327035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4986.041030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.077851] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.168028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4986.187134] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4987.479134] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4987.998763] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4991.464431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4991.476909] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4991.495540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4992.795703] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4993.374650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4995.340776] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4996.047871] Lustre: Failing over lustre-MDT0000 [ 4996.282134] Lustre: server umount lustre-MDT0000 complete [ 4996.575873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4996.578150] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.578964] LustreError: 324735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.582824] Lustre: Skipped 2 previous similar messages [ 4996.588049] LustreError: 324735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5006.818215] LustreError: 326147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.824816] LustreError: 326147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5009.230309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5009.281116] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5009.377421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.403548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5010.703433] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5013.596337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5014.499788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5014.501991] Lustre: Skipped 2 previous similar messages [ 5014.508951] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5014.524554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5015.670128] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5016.211093] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5019.616654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5019.617902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.618758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5019.618762] Lustre: Skipped 1 previous similar message [ 5019.625848] Lustre: Skipped 2 previous similar messages [ 5024.529851] Lustre: server umount lustre-MDT0000 complete [ 5024.736816] LustreError: 327035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.741206] LustreError: 327035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5025.677910] LustreError: 328253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444371 with bad export cookie 6305784624038749647 [ 5025.680516] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5025.680992] LustreError: 328253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5025.789202] Lustre: server umount lustre-MDT0001 complete [ 5037.273492] Lustre: server umount lustre-OST0000 complete [ 5042.401234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.590651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5043.914963] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5046.762631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.252985] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5049.967722] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5051.192847] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5053.558750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5055.568864] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5057.295732] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5066.981134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5072.354507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.354754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5072.355796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5072.355801] Lustre: Skipped 2 previous similar messages [ 5072.358723] Lustre: Skipped 1 previous similar message [ 5072.361277] LustreError: Skipped 1 previous similar message [ 5075.153378] Lustre: server umount lustre-OST0000 complete [ 5082.578930] Lustre: server umount lustre-MDT0000 complete [ 5082.592843] LustreError: 330530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.597573] LustreError: 330530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5083.772203] LustreError: 330493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444429 with bad export cookie 6305784624038750368 [ 5083.774771] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5083.776197] LustreError: 330493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5083.880021] Lustre: server umount lustre-MDT0001 complete [ 5087.220108] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5088.204754] Key type lgssc unregistered [ 5088.324331] LNet: 333707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5088.326725] LNetError: 333707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5088.335267] LNet: Removed LNI 192.168.202.130@tcp [ 5088.608958] Key type .llcrypt unregistered [ 5088.610164] Key type ._llcrypt unregistered [ 5098.230422] Key type ._llcrypt registered [ 5098.231537] Key type .llcrypt registered [ 5098.270957] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5098.676394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5098.729988] alg: No test for adler32 (adler32-zlib) [ 5099.599921] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5099.694410] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5101.287148] Key type lgssc registered [ 5101.618187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5104.090360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5104.096018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.258130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.563499] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5109.577135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.727825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5111.156182] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5112.960695] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5114.140123] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5116.392308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5116.515204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.390739] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5120.178904] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5129.830264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5135.329920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5135.330659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5135.331282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5135.333269] Lustre: Skipped 2 previous similar messages [ 5140.449460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5140.451889] Lustre: Skipped 4 previous similar messages [ 5140.826111] Lustre: server umount lustre-MDT0000 complete [ 5142.032428] LustreError: 337446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444487 with bad export cookie 10423718048142282813 [ 5142.036254] LustreError: 337446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5142.036276] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5142.147206] Lustre: server umount lustre-MDT0001 complete [ 5153.627617] Lustre: server umount lustre-OST0000 complete [ 5157.843167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.979682] LustreError: 338576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.986883] LustreError: 338576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5158.002643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.170077] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5161.657021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.937497] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5164.612565] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5165.798949] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5167.895455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5168.002901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.005163] Lustre: Skipped 1 previous similar message [ 5169.063544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5169.767374] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5171.490983] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5182.435114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.436431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5182.438593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5182.438917] Lustre: Skipped 2 previous similar messages [ 5184.274140] Lustre: server umount lustre-MDT0000 complete [ 5185.413448] LustreError: 338555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444530 with bad export cookie 10423718048142283912 [ 5185.415170] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5185.416605] LustreError: 338555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5185.517871] Lustre: server umount lustre-MDT0001 complete [ 5196.951845] Lustre: server umount lustre-OST0000 complete [ 5201.769563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.920646] LustreError: 342147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.940740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.204176] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5205.945606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.294873] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5208.876608] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5210.017426] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5212.183510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5214.055947] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5215.725472] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5226.660553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5237.217247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.218408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5237.221712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5237.224373] LustreError: Skipped 1 previous similar message [ 5237.227336] Lustre: Skipped 3 previous similar messages [ 5238.606510] Lustre: server umount lustre-OST0000 complete [ 5241.824343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5241.827528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.832245] Lustre: Skipped 1 previous similar message [ 5241.835396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.107043] Lustre: server umount lustre-MDT0000 complete [ 5247.329108] LustreError: 343761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444592 with bad export cookie 10423718048142284920 [ 5247.331227] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5247.333126] LustreError: 343761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5247.446495] Lustre: server umount lustre-MDT0001 complete [ 5251.048996] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5252.053699] Key type lgssc unregistered [ 5252.186284] LNet: 345347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5252.188514] LNetError: 345347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5252.197536] LNet: Removed LNI 192.168.202.130@tcp [ 5252.484242] Key type .llcrypt unregistered [ 5252.485545] Key type ._llcrypt unregistered [ 5261.559355] Key type ._llcrypt registered [ 5261.560585] Key type .llcrypt registered [ 5261.592062] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5261.920764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.963966] alg: No test for adler32 (adler32-zlib) [ 5262.831436] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5262.918164] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5264.495143] Key type lgssc registered [ 5264.906861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5267.579193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5267.584183] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.586822] LustreError: 346748:0:(osd_handler.c:8556:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5267.596075] LustreError: 346748:0:(obd_config.c:880:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5267.598904] LustreError: 346748:0:(obd_mount.c:217:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5267.601205] LustreError: 346748:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5267.603613] LustreError: 346748:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5270.065131] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5270.067474] LustreError: 347145:0:(osd_handler.c:8556:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5270.074747] LustreError: 347145:0:(obd_config.c:880:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5270.076664] LustreError: 347145:0:(obd_mount.c:217:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5270.078593] LustreError: 347145:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5270.080832] LustreError: 347145:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5275.811156] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5276.673614] Key type lgssc unregistered [ 5276.790463] LNet: 348130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5276.793379] LNetError: 348130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5276.806253] LNet: Removed LNI 192.168.202.130@tcp [ 5277.059722] Key type .llcrypt unregistered [ 5277.060605] Key type ._llcrypt unregistered [ 5282.665176] Key type ._llcrypt registered [ 5282.666170] Key type .llcrypt registered [ 5282.701764] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 5288.828742] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5289.342957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5289.351526] alg: No test for adler32 (adler32-zlib) [ 5290.209378] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5290.294173] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5291.879125] Key type lgssc registered [ 5292.280875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5304.002583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5304.008048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.099499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5305.114446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5305.150513] Lustre: lustre-MDT0000: new disk, initializing [ 5305.176407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.186526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5306.490710] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5310.678132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.726108] Lustre: 351224:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5310.752970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5310.754489] Lustre: Skipped 1 previous similar message [ 5310.787480] Lustre: lustre-MDT0001: new disk, initializing [ 5310.819877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5310.833547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5310.836941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5312.048163] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5315.082279] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5316.390544] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5318.694598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5318.808105] Lustre: lustre-OST0000: new disk, initializing [ 5318.810929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5318.841348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.776700] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5321.527570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5321.532608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5321.561149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5323.870143] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5325.236575] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5325.311200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5326.600649] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5326.665915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5331.937207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.938396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5331.939320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5331.940192] Lustre: Skipped 1 previous similar message [ 5331.946651] LustreError: Skipped 1 previous similar message [ 5333.397789] Lustre: server umount lustre-OST0000 complete [ 5336.543828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5336.546152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5336.551810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5336.553330] Lustre: Skipped 1 previous similar message [ 5342.176055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5342.178038] Lustre: Skipped 3 previous similar messages [ 5347.296107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.297879] Lustre: Skipped 1 previous similar message [ 5349.343125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5349.404050] Lustre: server umount lustre-MDT0000 complete [ 5350.686411] LustreError: 352964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444696 with bad export cookie 16337448897444457900 [ 5350.689629] LustreError: 352964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5350.689822] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.815279] Lustre: server umount lustre-MDT0001 complete [ 5353.704460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5354.265300] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:58:18 (1771444698) [ 5356.410786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.570322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.750982] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5360.440079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.804154] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5363.481670] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.603501] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5366.826578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5366.932338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5366.934750] Lustre: Skipped 1 previous similar message [ 5368.821422] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5370.654483] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5372.888043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5372.958492] Lustre: lustre-OST0001: new disk, initializing [ 5372.960420] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5374.874304] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5378.057020] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5379.067486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5379.070450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5379.099486] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5380.901542] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.903957] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5380.909542] Lustre: Skipped 2 previous similar messages [ 5385.986101] Lustre: server umount lustre-OST0001 complete [ 5386.211245] LustreError: 356871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.216988] LustreError: 356871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5391.336566] LustreError: 356871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5391.341482] LustreError: 356871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5396.449666] LustreError: 357377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.457962] LustreError: 357377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5401.055898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5401.059026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5401.063728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5401.065469] Lustre: Skipped 2 previous similar messages [ 5401.574133] LustreError: 357377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.579546] LustreError: 357377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5404.245127] Lustre: server umount lustre-OST0000 complete [ 5406.688862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5406.689106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.691448] LustreError: Skipped 1 previous similar message [ 5406.697266] Lustre: Skipped 1 previous similar message [ 5411.738468] Lustre: server umount lustre-MDT0000 complete [ 5411.807934] LustreError: 355299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5411.813808] LustreError: 355299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5412.970548] LustreError: 355262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444758 with bad export cookie 16337448897444458768 [ 5412.971356] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.974379] LustreError: 355262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5419.124795] Lustre: server umount lustre-MDT0001 complete [ 5422.522806] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5423.496036] Key type lgssc unregistered [ 5423.616397] LNet: 359630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5423.619228] LNetError: 359630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5423.632301] LNet: Removed LNI 192.168.202.130@tcp [ 5423.906870] Key type .llcrypt unregistered [ 5423.907957] Key type ._llcrypt unregistered [ 5440.873109] Key type ._llcrypt registered [ 5440.873926] Key type .llcrypt registered [ 5440.905232] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5441.304789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5441.312492] alg: No test for adler32 (adler32-zlib) [ 5442.166668] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5442.245890] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5443.823173] Key type lgssc registered [ 5444.164795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5446.565352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5446.572094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.655129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5447.662088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5447.719988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.892355] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5451.771414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.800681] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5451.810421] Lustre: 362387:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5451.892212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5453.043499] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5454.709220] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5455.894172] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5457.377171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5457.377309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.377309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.378532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5462.495870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5462.498158] Lustre: Skipped 2 previous similar messages [ 5462.732116] Lustre: server umount lustre-MDT0000 complete [ 5463.833939] LustreError: 363110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444809 with bad export cookie 1319997415592890956 [ 5463.837633] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.838181] LustreError: 363110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.929218] Lustre: server umount lustre-MDT0001 complete [ 5466.608379] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:00:10 (1771444810) [ 5468.594305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.753306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.976974] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5472.568757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.675321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5473.949702] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5475.598926] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5476.629910] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5478.610342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5478.671113] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5478.704809] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5478.745024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5480.447828] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5482.270754] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5491.285766] Lustre: Permanently deactivating lustre-OST0000 [ 5491.291972] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5495.087215] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5495.774683] LustreError: 367270:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5495.780410] LustreError: 367270:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5495.783957] LustreError: 367270:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5496.428610] LustreError: 367318:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5496.431100] LustreError: 367318:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5503.185408] Lustre: server umount lustre-OST0000 complete [ 5504.131120] LustreError: 367567:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5504.133384] LustreError: 367567:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5508.575812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5508.578984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.584485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.601039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.720297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5514.722401] Lustre: Skipped 3 previous similar messages [ 5519.327221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5519.391879] Lustre: server umount lustre-MDT0000 complete [ 5519.840121] LustreError: 364665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.845577] LustreError: 364665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5520.653416] LustreError: 364689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444866 with bad export cookie 1319997415592891453 [ 5520.658588] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5531.155210] Lustre: server umount lustre-MDT0001 complete [ 5534.131969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.539777] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5536.529109] LustreError: 368719:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5538.749209] Lustre: 369057:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.130@tcp [ 5539.503371] Lustre: server umount MGS complete [ 5543.161033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.302336] LustreError: 369783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.306580] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5543.309203] Lustre: Skipped 1 previous similar message [ 5543.327558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5544.438912] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5546.939267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.021538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5548.221337] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5549.846391] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5550.913362] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5551.504368] Lustre: Permanently reactivating lustre-OST0000 [ 5551.510471] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5551.512873] Lustre: Skipped 2 previous similar messages [ 5580.255119] LustreError: 371480:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5583.478503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.587070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.589037] Lustre: Skipped 1 previous similar message [ 5585.403444] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5587.241429] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5600.227712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.228195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.230724] LustreError: Skipped 1 previous similar message [ 5600.235638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5600.238263] Lustre: Skipped 1 previous similar message [ 5601.170109] Lustre: server umount lustre-OST0000 complete [ 5603.295709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.437293] Lustre: server umount lustre-MDT0000 complete [ 5605.343748] LustreError: 369801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.344471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.350592] Lustre: Skipped 2 previous similar messages [ 5605.556207] LustreError: 369762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771444950 with bad export cookie 1319997415592892475 [ 5605.559052] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.561435] LustreError: 369762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5605.668465] Lustre: server umount lustre-MDT0001 complete [ 5608.974362] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5609.897340] Key type lgssc unregistered [ 5610.031345] LNet: 374243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5610.033185] LNetError: 374243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5610.042329] LNet: Removed LNI 192.168.202.130@tcp [ 5610.324350] Key type .llcrypt unregistered [ 5610.325718] Key type ._llcrypt unregistered [ 5616.353522] Key type ._llcrypt registered [ 5616.354531] Key type .llcrypt registered [ 5616.389102] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 5622.808642] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5623.200493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5623.226339] alg: No test for adler32 (adler32-zlib) [ 5624.089503] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5624.176238] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5625.759227] Key type lgssc registered [ 5626.103410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5636.670192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5636.675298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.763134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5637.774432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5637.809485] Lustre: lustre-MDT0000: new disk, initializing [ 5637.833358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.841062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5639.044758] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5643.001720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.049568] Lustre: 377339:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5643.076395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5643.078433] Lustre: Skipped 1 previous similar message [ 5643.119323] Lustre: lustre-MDT0001: new disk, initializing [ 5643.150670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5643.163354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5643.166675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5644.361470] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5647.260034] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5648.384932] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5650.433042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5650.522871] Lustre: lustre-OST0000: new disk, initializing [ 5650.524818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5650.547127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.020977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5652.024153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5652.055678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5652.233623] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5655.040014] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5656.266530] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5656.341519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5657.488438] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5657.546828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5662.177599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5662.178041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.181442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5662.184983] Lustre: Skipped 1 previous similar message [ 5662.188419] Lustre: Skipped 1 previous similar message [ 5664.211063] Lustre: server umount lustre-OST0000 complete [ 5667.296899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.297455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.302110] Lustre: Skipped 1 previous similar message [ 5671.582454] Lustre: server umount lustre-MDT0000 complete [ 5672.853297] LustreError: 379082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445018 with bad export cookie 8887281905847419268 [ 5672.856358] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.857454] LustreError: 379082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5672.980729] Lustre: server umount lustre-MDT0001 complete [ 5675.832812] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5676.320860] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:03:40 (1771445020) [ 5678.430499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.565986] LustreError: 381391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.571022] LustreError: 381391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5678.586152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.725916] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5682.231277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.465928] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5684.990171] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5686.062224] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5688.184445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.330268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5688.333537] Lustre: Skipped 1 previous similar message [ 5690.342841] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5692.023559] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5692.602336] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5694.001162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5697.508021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.508055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.508682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.508686] Lustre: Skipped 1 previous similar message [ 5697.511423] Lustre: Skipped 1 previous similar message [ 5697.513374] LustreError: Skipped 1 previous similar message [ 5701.254762] Lustre: server umount lustre-OST0000 complete [ 5702.624246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.625030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5702.628387] Lustre: Skipped 1 previous similar message [ 5702.630729] Lustre: Skipped 1 previous similar message [ 5707.744743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.746658] Lustre: Skipped 2 previous similar messages [ 5708.503912] Lustre: server umount lustre-MDT0000 complete [ 5709.710125] LustreError: 381371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445055 with bad export cookie 8887281905847420136 [ 5709.713153] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5709.715179] LustreError: 381371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5709.830630] Lustre: server umount lustre-MDT0001 complete [ 5713.541786] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5714.555284] Key type lgssc unregistered [ 5714.685356] LNet: 384758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5714.689094] LNetError: 384758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5714.696547] LNet: Removed LNI 192.168.202.130@tcp [ 5714.998188] Key type .llcrypt unregistered [ 5714.999275] Key type ._llcrypt unregistered [ 5722.743142] Key type ._llcrypt registered [ 5722.744569] Key type .llcrypt registered [ 5722.787552] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5722.905485] Key type .llcrypt unregistered [ 5722.906785] Key type ._llcrypt unregistered [ 5729.956262] Key type ._llcrypt registered [ 5729.957324] Key type .llcrypt registered [ 5729.989773] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5730.368726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5730.430170] alg: No test for adler32 (adler32-zlib) [ 5731.296356] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5731.399857] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5732.983120] Key type lgssc registered [ 5733.333173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5736.013592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5736.020751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5737.200378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.580826] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5741.419917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5741.531654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.436075] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5745.187356] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5746.658683] LustreError: 386911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5746.663283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5746.666797] LustreError: 386911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5747.558179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.703049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.055648] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5756.895771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5756.898445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.902497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5757.923086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5757.926813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5757.927422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.320725] Lustre: server umount lustre-OST0000 complete [ 5767.137405] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5767.139960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.143872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5767.145614] Lustre: Skipped 1 previous similar message [ 5769.749059] Lustre: server umount lustre-MDT0000 complete [ 5770.996190] LustreError: 388524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445116 with bad export cookie 11711379362909434005 [ 5771.000171] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5771.002559] LustreError: 388524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5771.129241] Lustre: server umount lustre-MDT0001 complete [ 5774.917732] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5775.996449] Key type lgssc unregistered [ 5776.137594] LNet: 389897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5776.140783] LNetError: 389897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5776.151454] LNet: Removed LNI 192.168.202.130@tcp [ 5776.423097] Key type .llcrypt unregistered [ 5776.424147] Key type ._llcrypt unregistered [ 5785.385783] Key type ._llcrypt registered [ 5785.386902] Key type .llcrypt registered [ 5785.423560] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5785.867960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5785.931957] alg: No test for adler32 (adler32-zlib) [ 5786.779514] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5786.860636] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5788.439129] Key type lgssc registered [ 5788.789092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5791.443349] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5791.450409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5806.879348] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5811.999274] LustreError: 391246:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 5838.623252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.622411] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5842.528145] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5844.907934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.082172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.107131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5846.387771] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5849.265497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.393386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.711154] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5852.473834] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5853.679808] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5858.785900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.790710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.792723] Lustre: Skipped 1 previous similar message [ 5862.095340] Lustre: server umount lustre-OST0000 complete [ 5863.904825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.905584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5863.908669] Lustre: Skipped 2 previous similar messages [ 5869.024398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.028173] Lustre: Skipped 2 previous similar messages [ 5869.535857] Lustre: server umount lustre-MDT0000 complete [ 5870.912463] LustreError: 391252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445216 with bad export cookie 15052413748805982226 [ 5870.915766] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5870.916103] LustreError: 391252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5871.055231] Lustre: server umount lustre-MDT0001 complete [ 5875.046899] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5876.159943] Key type lgssc unregistered [ 5876.304420] LNet: 394458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5876.307131] LNetError: 394458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5876.315478] LNet: Removed LNI 192.168.202.130@tcp [ 5876.667201] Key type .llcrypt unregistered [ 5876.668613] Key type ._llcrypt unregistered [ 5885.415454] Key type ._llcrypt registered [ 5885.416268] Key type .llcrypt registered [ 5885.450340] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5885.882342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5885.888644] alg: No test for adler32 (adler32-zlib) [ 5886.736950] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5886.820301] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5888.399179] Key type lgssc registered [ 5888.740934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5891.124994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5891.131452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.280285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.447767] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5895.997535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5896.102736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5897.285319] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5898.865742] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5899.880453] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5901.822873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.913300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.954125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5903.787214] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5905.645563] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5911.522272] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.524136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5911.524196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.528354] Lustre: Skipped 2 previous similar messages [ 5912.165405] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 5912.169329] Lustre: Skipped 2 previous similar messages [ 5913.623713] Lustre: server umount lustre-MDT0000 complete [ 5914.368653] LustreError: 395815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.30@tcp arrived at 1771445259 with bad export cookie 7510661579317462947 [ 5914.372342] LustreError: 395815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5914.993996] LustreError: 395815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445260 with bad export cookie 7510661579317462100 [ 5914.997963] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5916.640131] LustreError: 396546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.640505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.644306] LustreError: 396546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5916.646371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.654431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.647926] LustreError: 396546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.648334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.653395] LustreError: 396546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5921.106065] Lustre: server umount lustre-OST0000 complete [ 5923.304516] Lustre: server umount lustre-MDT0001 complete [ 5926.829041] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5927.848558] Key type lgssc unregistered [ 5927.980329] LNet: 399165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5927.982724] LNetError: 399165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5927.994298] LNet: Removed LNI 192.168.202.130@tcp [ 5928.343137] Key type .llcrypt unregistered [ 5928.344957] Key type ._llcrypt unregistered [ 5937.335156] Key type ._llcrypt registered [ 5937.336286] Key type .llcrypt registered [ 5937.375269] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5937.699515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5937.717349] alg: No test for adler32 (adler32-zlib) [ 5938.569729] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 5938.650173] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5940.231101] Key type lgssc registered [ 5940.581850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5943.192957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5943.198372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.345862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.561904] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5948.379542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.511789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5949.787478] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5951.523485] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5952.724644] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5955.001207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5955.115675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.030666] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5958.782336] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5969.446374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5974.497542] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5974.498623] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.499482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5974.506224] Lustre: Skipped 1 previous similar message [ 5976.465585] Lustre: server umount lustre-MDT0001 complete [ 5979.616603] LustreError: 400541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.617388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5979.617530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.618541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5979.618545] Lustre: Skipped 2 previous similar messages [ 5979.621306] LustreError: 400541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5984.271557] Lustre: server umount lustre-OST0000 complete [ 5984.736314] LustreError: 400541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5985.672541] Lustre: server umount lustre-MDT0000 complete [ 5990.159350] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5991.226373] Key type lgssc unregistered [ 5991.361400] LNet: 403888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5991.364623] LNetError: 403888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5991.374343] LNet: Removed LNI 192.168.202.130@tcp [ 5991.677909] Key type .llcrypt unregistered [ 5991.679082] Key type ._llcrypt unregistered [ 5999.723312] Key type ._llcrypt registered [ 5999.724394] Key type .llcrypt registered [ 5999.762727] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 6005.737803] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6006.130088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6006.149624] alg: No test for adler32 (adler32-zlib) [ 6007.002348] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 6007.086285] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6008.671117] Key type lgssc registered [ 6009.032754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6020.849087] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6025.857691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6025.863305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.952618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6026.964521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6027.004296] Lustre: lustre-MDT0000: new disk, initializing [ 6027.030864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.038384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6028.274270] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6033.882462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.985261] Lustre: lustre-OST0000: new disk, initializing [ 6033.987790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6033.989393] Lustre: Skipped 1 previous similar message [ 6034.009815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.945383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6034.948970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6034.959225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6035.845572] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6038.867895] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6044.152643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.195192] Lustre: 407825:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6044.233815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6044.247480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6044.282650] Lustre: lustre-MDT0001: new disk, initializing [ 6044.323326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6044.336499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6044.340758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6045.639912] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6049.097628] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6049.164759] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6050.433742] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6050.503345] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6053.476054] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 6054.884366] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.885061] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6054.888358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6054.893127] Lustre: Skipped 1 previous similar message [ 6058.132158] Lustre: server umount lustre-MDT0001 complete [ 6058.588858] LustreError: 407831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.595592] LustreError: 407831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6060.001716] LustreError: 408535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.007603] LustreError: 408535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6065.120861] LustreError: 411236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.124840] LustreError: 411236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6070.241745] LustreError: 407831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.247740] LustreError: 407831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6070.751840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.754201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.757910] Lustre: Skipped 1 previous similar message [ 6070.759829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6075.360258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6075.360789] LustreError: 410049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.498908] Lustre: server umount lustre-OST0000 complete [ 6092.255144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.365062] Lustre: server umount lustre-MDT0000 complete [ 6096.689679] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6097.689729] Key type lgssc unregistered [ 6097.812371] LNet: 412393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6097.814559] LNetError: 412393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6097.826314] LNet: Removed LNI 192.168.202.130@tcp [ 6098.131504] Key type .llcrypt unregistered [ 6098.132865] Key type ._llcrypt unregistered [ 6110.328334] Key type ._llcrypt registered [ 6110.329373] Key type .llcrypt registered [ 6110.364540] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6110.465208] Key type .llcrypt unregistered [ 6110.467787] Key type ._llcrypt unregistered [ 6116.987328] Key type ._llcrypt registered [ 6116.988922] Key type .llcrypt registered [ 6117.035041] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6117.505693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.573127] alg: No test for adler32 (adler32-zlib) [ 6118.451728] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 6118.551388] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6120.135136] Key type lgssc registered [ 6120.505238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6129.167073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6129.172466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.263036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6130.273932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6130.312229] Lustre: lustre-MDT0000: new disk, initializing [ 6130.335506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.341979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6131.626194] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6135.622574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.667255] Lustre: 415708:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6135.691721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6135.694510] Lustre: Skipped 1 previous similar message [ 6135.733395] Lustre: lustre-MDT0001: new disk, initializing [ 6135.767957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6135.780199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6135.783387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6136.990377] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6139.963664] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6141.074472] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6143.201571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6143.317271] Lustre: lustre-OST0000: new disk, initializing [ 6143.319858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6143.322499] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6143.361995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.250242] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6148.241829] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6148.584149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6148.588695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6148.600536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6156.255909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6156.259428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.263705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.817069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6158.817903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.824162] Lustre: Skipped 1 previous similar message [ 6162.011860] Lustre: server umount lustre-MDT0000 complete [ 6163.261967] LustreError: 415698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445508 with bad export cookie 7443537013602526515 [ 6163.265587] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6163.266647] LustreError: 415698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.380734] Lustre: server umount lustre-MDT0001 complete [ 6174.873868] Lustre: server umount lustre-OST0000 complete [ 6178.266647] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:12:02 (1771445522) [ 6180.771857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.940700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.182846] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6184.961636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.081314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6186.360342] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6188.178633] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6189.407555] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6202.741742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6202.822525] Lustre: Found index 0 for lustre-OST0000, updating log [ 6202.882809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6204.773829] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6206.494757] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6207.014465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6215.138090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6215.138805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.140909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.143538] Lustre: Skipped 1 previous similar message [ 6215.146629] Lustre: Skipped 3 previous similar messages [ 6218.769648] Lustre: server umount lustre-OST0000 complete [ 6231.519835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6231.522177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6231.526150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6231.528436] Lustre: Skipped 2 previous similar messages [ 6237.399033] Lustre: server umount lustre-MDT0000 complete [ 6237.664066] LustreError: 419367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.671691] LustreError: 419367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6238.602219] LustreError: 419349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771445584 with bad export cookie 7443537013602527908 [ 6238.605776] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6238.608509] LustreError: 419349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6238.723508] Lustre: server umount lustre-MDT0001 complete [ 6241.398200] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:13:05 (1771445585) [ 6267.039277] LNet: 1 local NIs in recovery (showing 1): 192.168.202.130@tcp [ 6820.778176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.939289] LustreError: 425432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.963706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.220742] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6825.005259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.531725] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6828.295915] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6829.480034] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6842.678242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6842.775720] Lustre: Found index 0 for lustre-OST0000, updating log [ 6844.709749] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6846.467774] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6847.176133] Lustre: server umount lustre-OST0000 complete [ 6851.039795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6851.042160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.046397] Lustre: Skipped 1 previous similar message [ 6851.049150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.051253] Lustre: Skipped 2 previous similar messages [ 6852.065497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6852.065935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.552566] Lustre: server umount lustre-MDT0000 complete [ 6855.844325] LustreError: 425412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771446201 with bad export cookie 7443537013602528979 [ 6855.848077] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6855.849275] LustreError: 425412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6855.971185] Lustre: server umount lustre-MDT0001 complete [ 6873.042323] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 6875.927346] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6881.440168] LNet: 1 local NIs in recovery (showing 1): 192.168.202.130@tcp [ 6887.518788] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 15:23:51 (1771446231) [ 6893.296409] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6894.285318] Key type lgssc unregistered [ 6905.696215] LNet: 431689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6905.700538] LNetError: 431689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6973.285337] LNet: Removed LNI 192.168.202.130@tcp [ 6973.571415] Key type .llcrypt unregistered [ 6973.572411] Key type ._llcrypt unregistered [ 6979.937225] Key type ._llcrypt registered [ 6979.938879] Key type .llcrypt registered [ 6979.977548] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6980.411327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6980.444535] alg: No test for adler32 (adler32-zlib) [ 6981.292268] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 6981.377817] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6982.959114] Key type lgssc registered [ 6983.286918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6984.608961] LNet: Added LNI 192.168.202.130@tcp100 [8/256/0/180] [ 6984.901439] LNet: Added LNI 192.168.202.130@tcp101 [8/256/0/180] [ 6985.177439] LNet: Added LNI 192.168.202.130@tcp102 [8/256/0/180] [ 6985.495325] LNet: Added LNI 192.168.202.130@tcp103 [8/256/0/180] [ 6985.805491] LNet: Added LNI 192.168.202.130@tcp104 [8/256/0/180] [ 6986.113756] LNet: Added LNI 192.168.202.130@tcp105 [8/256/0/180] [ 6986.415482] LNet: Added LNI 192.168.202.130@tcp106 [8/256/0/180] [ 6986.701267] LNet: Added LNI 192.168.202.130@tcp107 [8/256/0/180] [ 6986.995197] LNet: Added LNI 192.168.202.130@tcp108 [8/256/0/180] [ 6987.271688] LNet: Added LNI 192.168.202.130@tcp109 [8/256/0/180] [ 6987.548933] LNet: Added LNI 192.168.202.130@tcp110 [8/256/0/180] [ 6987.830667] LNet: Added LNI 192.168.202.130@tcp111 [8/256/0/180] [ 6988.118777] LNet: Added LNI 192.168.202.130@tcp112 [8/256/0/180] [ 6988.394939] LNet: Added LNI 192.168.202.130@tcp113 [8/256/0/180] [ 6988.686443] LNet: Added LNI 192.168.202.130@tcp114 [8/256/0/180] [ 6988.979758] LNet: Added LNI 192.168.202.130@tcp115 [8/256/0/180] [ 6989.279661] LNet: Added LNI 192.168.202.130@tcp116 [8/256/0/180] [ 6989.569683] LNet: Added LNI 192.168.202.130@tcp117 [8/256/0/180] [ 6989.852646] LNet: Added LNI 192.168.202.130@tcp118 [8/256/0/180] [ 6990.151901] LNet: Added LNI 192.168.202.130@tcp119 [8/256/0/180] [ 6990.448137] LNet: Added LNI 192.168.202.130@tcp120 [8/256/0/180] [ 6990.730486] LNet: Added LNI 192.168.202.130@tcp121 [8/256/0/180] [ 6991.020175] LNet: Added LNI 192.168.202.130@tcp122 [8/256/0/180] [ 6991.306918] LNet: Added LNI 192.168.202.130@tcp123 [8/256/0/180] [ 6991.584421] LNet: Added LNI 192.168.202.130@tcp124 [8/256/0/180] [ 6991.863456] LNet: Added LNI 192.168.202.130@tcp125 [8/256/0/180] [ 6992.149498] LNet: Added LNI 192.168.202.130@tcp126 [8/256/0/180] [ 6992.433316] LNet: Added LNI 192.168.202.130@tcp127 [8/256/0/180] [ 6992.712754] LNet: Added LNI 192.168.202.130@tcp128 [8/256/0/180] [ 6993.000436] LNet: Added LNI 192.168.202.130@tcp129 [8/256/0/180] [ 6993.293457] LNet: Added LNI 192.168.202.130@tcp130 [8/256/0/180] [ 6993.593975] LNet: Added LNI 192.168.202.130@tcp131 [8/256/0/180] [ 6993.888525] LNet: Added LNI 192.168.202.130@tcp132 [8/256/0/180] [ 6994.180695] LNet: Added LNI 192.168.202.130@tcp133 [8/256/0/180] [ 6994.471626] LNet: Added LNI 192.168.202.130@tcp134 [8/256/0/180] [ 6994.766181] LNet: Added LNI 192.168.202.130@tcp135 [8/256/0/180] [ 6996.903956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6996.910633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.961033] Lustre: 434670:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [ 6998.036710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6998.049354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6998.084722] Lustre: lustre-MDT0000: new disk, initializing [ 6998.115294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6998.121667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6999.322541] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7003.376462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.442719] Lustre: 434689:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7003.472303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7003.474010] Lustre: Skipped 1 previous similar message [ 7003.513495] Lustre: lustre-MDT0001: new disk, initializing [ 7003.526738] Lustre: 435471:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [ 7003.529336] Lustre: 435471:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7003.587386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7003.603142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7003.608842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7004.872774] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7006.643259] Lustre: 435843:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [ 7006.646063] Lustre: 435843:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7007.918836] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7009.192318] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7011.357927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.468798] Lustre: lustre-OST0000: new disk, initializing [ 7011.471404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7011.508791] Lustre: 436440:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [ 7011.514565] Lustre: 436440:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7011.535068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7013.290471] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7015.484095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7015.486967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7015.512950] Lustre: 436948:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [ 7015.517657] Lustre: 436948:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7015.548205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7016.305940] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7020.511805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7020.514754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7020.518882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.025262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.027072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7021.028780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.589422] Lustre: server umount lustre-OST0000 complete [ 7026.144902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7026.145539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7026.148465] Lustre: Skipped 1 previous similar message [ 7026.151331] Lustre: Skipped 1 previous similar message [ 7031.264608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7031.266505] Lustre: Skipped 2 previous similar messages [ 7031.903651] Lustre: server umount lustre-MDT0000 complete [ 7033.158175] LustreError: 436445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771446378 with bad export cookie 13450060468524674379 [ 7033.161119] LustreError: 436445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7033.161857] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7033.278336] Lustre: server umount lustre-MDT0001 complete [ 7036.923094] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7037.978163] Key type lgssc unregistered [ 7038.141776] LNet: 438214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7038.144739] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7038.146801] LNet: Removed LNI 192.168.202.130@tcp [ 7039.199131] LNet: Removed LNI 192.168.202.130@tcp100 [ 7039.201108] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7039.204810] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7039.208281] LNet: Removed LNI 192.168.202.130@tcp101 [ 7040.223157] LNet: Removed LNI 192.168.202.130@tcp102 [ 7040.224530] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7040.226599] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7040.228886] LNet: Removed LNI 192.168.202.130@tcp103 [ 7041.247157] LNet: Removed LNI 192.168.202.130@tcp104 [ 7042.271159] LNet: Removed LNI 192.168.202.130@tcp105 [ 7042.273324] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7042.277098] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7042.281174] LNet: Removed LNI 192.168.202.130@tcp106 [ 7043.295108] LNet: Removed LNI 192.168.202.130@tcp107 [ 7044.319116] LNet: Removed LNI 192.168.202.130@tcp108 [ 7045.343208] LNet: Removed LNI 192.168.202.130@tcp109 [ 7046.367195] LNet: Removed LNI 192.168.202.130@tcp110 [ 7046.369379] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7046.373196] LNetError: 438214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7046.377142] LNet: Removed LNI 192.168.202.130@tcp111 [ 7047.391134] LNet: Removed LNI 192.168.202.130@tcp112 [ 7048.415089] LNet: Removed LNI 192.168.202.130@tcp113 [ 7049.439134] LNet: Removed LNI 192.168.202.130@tcp114 [ 7049.440285] LNet: Removed LNI 192.168.202.130@tcp115 [ 7049.441325] LNet: Removed LNI 192.168.202.130@tcp116 [ 7049.442395] LNet: Removed LNI 192.168.202.130@tcp117 [ 7049.443415] LNet: Removed LNI 192.168.202.130@tcp118 [ 7049.444471] LNet: Removed LNI 192.168.202.130@tcp119 [ 7049.445519] LNet: Removed LNI 192.168.202.130@tcp120 [ 7049.446595] LNet: Removed LNI 192.168.202.130@tcp121 [ 7049.447654] LNet: Removed LNI 192.168.202.130@tcp122 [ 7049.448709] LNet: Removed LNI 192.168.202.130@tcp123 [ 7049.449770] LNet: Removed LNI 192.168.202.130@tcp124 [ 7049.450873] LNet: Removed LNI 192.168.202.130@tcp125 [ 7049.451948] LNet: Removed LNI 192.168.202.130@tcp126 [ 7049.453051] LNet: Removed LNI 192.168.202.130@tcp127 [ 7049.454115] LNet: Removed LNI 192.168.202.130@tcp128 [ 7049.455201] LNet: Removed LNI 192.168.202.130@tcp129 [ 7049.456275] LNet: Removed LNI 192.168.202.130@tcp130 [ 7049.457695] LNet: Removed LNI 192.168.202.130@tcp131 [ 7049.459113] LNet: Removed LNI 192.168.202.130@tcp132 [ 7049.460187] LNet: Removed LNI 192.168.202.130@tcp133 [ 7049.461224] LNet: Removed LNI 192.168.202.130@tcp134 [ 7049.468453] LNet: Removed LNI 192.168.202.130@tcp135 [ 7049.729664] Key type .llcrypt unregistered [ 7049.730714] Key type ._llcrypt unregistered [ 7056.815412] Key type ._llcrypt registered [ 7056.816535] Key type .llcrypt registered [ 7056.851574] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 7062.753744] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7063.079327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7063.129047] alg: No test for adler32 (adler32-zlib) [ 7063.995687] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 7064.075664] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7065.655124] Key type lgssc registered [ 7066.014928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7076.766810] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:27:00 (1771446420) [ 7082.439678] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7083.431135] Key type lgssc unregistered [ 7083.548326] LNet: 442219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7083.551500] LNetError: 442219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7083.561262] LNet: Removed LNI 192.168.202.130@tcp [ 7083.829367] Key type .llcrypt unregistered [ 7083.830911] Key type ._llcrypt unregistered [ 7090.384144] Key type ._llcrypt registered [ 7090.385178] Key type .llcrypt registered [ 7090.421704] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7090.814890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7090.871038] alg: No test for adler32 (adler32-zlib) [ 7091.744815] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 7091.846234] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7093.431160] Key type lgssc registered [ 7093.827369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7096.927877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7096.933305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.025605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7098.036744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7098.075212] Lustre: lustre-MDT0000: new disk, initializing [ 7098.098408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.104402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7099.342300] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7103.420438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.465384] Lustre: 443393:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7103.487578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7103.489310] Lustre: Skipped 1 previous similar message [ 7103.524243] Lustre: lustre-MDT0001: new disk, initializing [ 7103.553234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7103.564633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7103.568521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7104.833528] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7107.868067] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7109.039445] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7111.297137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7111.423204] Lustre: lustre-OST0000: new disk, initializing [ 7111.426667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7111.465370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.372561] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7114.805697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7114.808976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7114.837784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7116.359200] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7117.545826] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing lnet_if_list [ 7119.156788] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7119.169466] LNet: Added LNI 192.168.202.130@tcp10 [8/256/0/180] [ 7120.941614] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7120.951760] LNet: Added LNI 192.168.202.130@tcp11 [8/256/0/180] [ 7122.196036] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7122.205446] LNet: Added LNI 192.168.202.130@tcp12 [8/256/0/180] [ 7123.477062] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7123.491089] LNet: Added LNI 192.168.202.130@tcp13 [8/256/0/180] [ 7124.801336] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7124.819264] LNet: Added LNI 192.168.202.130@tcp14 [8/256/0/180] [ 7126.194186] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7126.208318] LNet: Added LNI 192.168.202.130@tcp15 [8/256/0/180] [ 7127.664462] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7127.674785] LNet: Added LNI 192.168.202.130@tcp16 [8/256/0/180] [ 7129.075592] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7129.085702] LNet: Added LNI 192.168.202.130@tcp17 [8/256/0/180] [ 7130.686641] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7130.697911] LNet: Added LNI 192.168.202.130@tcp18 [8/256/0/180] [ 7132.188720] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7132.199514] LNet: Added LNI 192.168.202.130@tcp19 [8/256/0/180] [ 7133.773681] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7133.782513] LNet: Added LNI 192.168.202.130@tcp20 [8/256/0/180] [ 7135.310807] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7135.322511] LNet: Added LNI 192.168.202.130@tcp21 [8/256/0/180] [ 7136.988618] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7137.004748] LNet: Added LNI 192.168.202.130@tcp22 [8/256/0/180] [ 7138.893825] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7138.904996] LNet: Added LNI 192.168.202.130@tcp23 [8/256/0/180] [ 7140.773737] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7140.785415] LNet: Added LNI 192.168.202.130@tcp24 [8/256/0/180] [ 7142.729179] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7142.740750] LNet: Added LNI 192.168.202.130@tcp25 [8/256/0/180] [ 7144.582642] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7144.594756] LNet: Added LNI 192.168.202.130@tcp26 [8/256/0/180] [ 7146.541640] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7146.551628] LNet: Added LNI 192.168.202.130@tcp27 [8/256/0/180] [ 7148.487678] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7148.497895] LNet: Added LNI 192.168.202.130@tcp28 [8/256/0/180] [ 7150.431726] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7150.443595] LNet: Added LNI 192.168.202.130@tcp29 [8/256/0/180] [ 7152.262789] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7152.274557] LNet: Added LNI 192.168.202.130@tcp30 [8/256/0/180] [ 7154.175335] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7154.186049] LNet: Added LNI 192.168.202.130@tcp31 [8/256/0/180] [ 7156.080569] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7156.094456] LNet: Added LNI 192.168.202.130@tcp32 [8/256/0/180] [ 7158.029949] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7158.042845] LNet: Added LNI 192.168.202.130@tcp33 [8/256/0/180] [ 7159.999185] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7160.011768] LNet: Added LNI 192.168.202.130@tcp34 [8/256/0/180] [ 7162.008739] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7162.020444] LNet: Added LNI 192.168.202.130@tcp35 [8/256/0/180] [ 7164.032334] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7164.042550] LNet: Added LNI 192.168.202.130@tcp36 [8/256/0/180] [ 7166.133590] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7166.142980] LNet: Added LNI 192.168.202.130@tcp37 [8/256/0/180] [ 7168.176475] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7168.186322] LNet: Added LNI 192.168.202.130@tcp38 [8/256/0/180] [ 7170.375252] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7170.384948] LNet: Added LNI 192.168.202.130@tcp39 [8/256/0/180] [ 7172.604656] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7172.613915] LNet: Added LNI 192.168.202.130@tcp40 [8/256/0/180] [ 7174.922591] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7174.937058] LNet: Added LNI 192.168.202.130@tcp41 [8/256/0/180] [ 7177.177046] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7177.187116] LNet: Added LNI 192.168.202.130@tcp42 [8/256/0/180] [ 7179.364341] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7179.375047] LNet: Added LNI 192.168.202.130@tcp43 [8/256/0/180] [ 7181.655111] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7181.665580] LNet: Added LNI 192.168.202.130@tcp44 [8/256/0/180] [ 7184.007254] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7184.016405] LNet: Added LNI 192.168.202.130@tcp45 [8/256/0/180] [ 7186.402986] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7186.413070] LNet: Added LNI 192.168.202.130@tcp46 [8/256/0/180] [ 7188.720433] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7188.729744] LNet: Added LNI 192.168.202.130@tcp47 [8/256/0/180] [ 7191.031208] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7191.041772] LNet: Added LNI 192.168.202.130@tcp48 [8/256/0/180] [ 7193.626770] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7193.640070] LNet: Added LNI 192.168.202.130@tcp49 [8/256/0/180] [ 7196.127539] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7196.142327] LNet: Added LNI 192.168.202.130@tcp50 [8/256/0/180] [ 7205.855872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7205.857878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.861310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.881673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.882901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.886331] Lustre: Skipped 1 previous similar message [ 7206.888401] Lustre: Skipped 1 previous similar message [ 7209.686382] Lustre: server umount lustre-MDT0000 complete [ 7210.948792] LustreError: 443383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771446556 with bad export cookie 9135627318097713082 [ 7210.956298] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7211.081399] Lustre: server umount lustre-MDT0001 complete [ 7217.938595] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7220.731247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.151358] LustreError: 442663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000369248dc x1857496219203456/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7221.276680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7222.511249] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7225.924751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.016474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.048983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7227.109982] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7227.269343] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7228.127083] Lustre: 442664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771446557/real 1771446557] req@00000000ae96e0ff x1857496219203200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771446573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7228.957885] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7230.077838] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7232.479343] Lustre: 442666:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771446561/real 1771446561] req@000000000db929f7 x1857496219203328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771446577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7236.575107] Lustre: 442667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771446566/real 1771446566] req@0000000015372f35 x1857496219203584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771446582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7237.604646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.615130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7237.619447] Lustre: Skipped 1 previous similar message [ 7242.721948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7242.724595] Lustre: Skipped 1 previous similar message [ 7243.153560] Lustre: server umount lustre-OST0000 complete [ 7246.815868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7246.817892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.821142] Lustre: Skipped 1 previous similar message [ 7246.822590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7250.456045] Lustre: server umount lustre-MDT0000 complete [ 7251.698335] LustreError: 443383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771446597 with bad export cookie 9135627318097715119 [ 7251.701344] LustreError: 443383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7251.701821] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.801824] Lustre: server umount lustre-MDT0001 complete [ 7255.028249] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7256.024497] Key type lgssc unregistered [ 7256.185627] LNet: 471052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7256.189824] LNetError: 471052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7256.194335] LNet: Removed LNI 192.168.202.130@tcp [ 7256.198244] LNet: Removed LNI 192.168.202.130@tcp10 [ 7256.202777] LNet: Removed LNI 192.168.202.130@tcp11 [ 7256.206841] LNet: Removed LNI 192.168.202.130@tcp12 [ 7256.209071] LNet: Removed LNI 192.168.202.130@tcp13 [ 7256.211340] LNet: Removed LNI 192.168.202.130@tcp14 [ 7256.214632] LNet: Removed LNI 192.168.202.130@tcp15 [ 7256.217622] LNet: Removed LNI 192.168.202.130@tcp16 [ 7256.219921] LNet: Removed LNI 192.168.202.130@tcp17 [ 7256.222836] LNet: Removed LNI 192.168.202.130@tcp18 [ 7256.225793] LNet: Removed LNI 192.168.202.130@tcp19 [ 7256.228106] LNet: Removed LNI 192.168.202.130@tcp20 [ 7256.231132] LNet: Removed LNI 192.168.202.130@tcp21 [ 7256.234680] LNet: Removed LNI 192.168.202.130@tcp22 [ 7256.238216] LNet: Removed LNI 192.168.202.130@tcp23 [ 7256.241756] LNet: Removed LNI 192.168.202.130@tcp24 [ 7256.245164] LNet: Removed LNI 192.168.202.130@tcp25 [ 7256.247691] LNet: Removed LNI 192.168.202.130@tcp26 [ 7256.249534] LNet: Removed LNI 192.168.202.130@tcp27 [ 7256.251345] LNet: Removed LNI 192.168.202.130@tcp28 [ 7256.253176] LNet: Removed LNI 192.168.202.130@tcp29 [ 7256.254928] LNet: Removed LNI 192.168.202.130@tcp30 [ 7256.256798] LNet: Removed LNI 192.168.202.130@tcp31 [ 7256.258575] LNet: Removed LNI 192.168.202.130@tcp32 [ 7256.260411] LNet: Removed LNI 192.168.202.130@tcp33 [ 7256.262314] LNet: Removed LNI 192.168.202.130@tcp34 [ 7256.264126] LNet: Removed LNI 192.168.202.130@tcp35 [ 7256.265883] LNet: Removed LNI 192.168.202.130@tcp36 [ 7256.267762] LNet: Removed LNI 192.168.202.130@tcp37 [ 7256.269555] LNet: Removed LNI 192.168.202.130@tcp38 [ 7256.271388] LNet: Removed LNI 192.168.202.130@tcp39 [ 7256.273208] LNet: Removed LNI 192.168.202.130@tcp40 [ 7256.274980] LNet: Removed LNI 192.168.202.130@tcp41 [ 7256.276827] LNet: Removed LNI 192.168.202.130@tcp42 [ 7256.279633] LNet: Removed LNI 192.168.202.130@tcp43 [ 7256.281668] LNet: Removed LNI 192.168.202.130@tcp44 [ 7256.283336] LNet: Removed LNI 192.168.202.130@tcp45 [ 7256.284975] LNet: Removed LNI 192.168.202.130@tcp46 [ 7256.286619] LNet: Removed LNI 192.168.202.130@tcp47 [ 7256.288299] LNet: Removed LNI 192.168.202.130@tcp48 [ 7256.289912] LNet: Removed LNI 192.168.202.130@tcp49 [ 7256.297391] LNet: Removed LNI 192.168.202.130@tcp50 [ 7256.572324] Key type .llcrypt unregistered [ 7256.573333] Key type ._llcrypt unregistered [ 7263.617610] Key type ._llcrypt registered [ 7263.618631] Key type .llcrypt registered [ 7263.651403] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7264.090769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7264.100270] alg: No test for adler32 (adler32-zlib) [ 7264.940168] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 7265.024599] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7266.607123] Key type lgssc registered [ 7266.958597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7269.622626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7269.628059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.780241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7272.102778] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7275.164552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.292299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7276.578192] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7278.379299] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7279.563693] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7282.211362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7282.381067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7284.902860] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7287.440952] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7296.746525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7301.817945] loop0: detected capacity change from 0 to 204800000 [ 7301.873946] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7301.876936] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7301.880462] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7303.827388] loop0: detected capacity change from 0 to 204800000 [ 7303.890924] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7303.893826] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7303.896822] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7305.608284] loop0: detected capacity change from 0 to 204800000 [ 7307.673080] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.718943] Lustre: fs15246-OST0000: new disk, initializing [ 7307.720623] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7307.736416] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.812468] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7314.252236] loop1: detected capacity change from 0 to 204800000 [ 7316.606142] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.660194] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7316.683471] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7316.726212] Lustre: fs15246-MDT0000: new disk, initializing [ 7316.775068] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7316.784700] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7316.823527] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7317.855691] Lustre: 476654:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000336c206f x1857496400560256/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1771446679 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7318.117397] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7326.943164] LustreError: 476654:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7326.946242] LustreError: 476654:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7326.949051] LustreError: 476654:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7328.739909] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.745316] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7331.600207] Lustre: server umount fs15246-MDT0000 complete [ 7333.859180] LustreError: 472271:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7333.865304] LustreError: 472271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7334.523024] Lustre: server umount fs15246-OST0000 complete [ 7338.978119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.979235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7338.982612] Lustre: Skipped 1 previous similar message [ 7344.097084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7344.099788] Lustre: Skipped 2 previous similar messages [ 7344.401141] Lustre: server umount lustre-OST0000 complete [ 7347.167906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7347.170077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.173851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7349.217942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.704187] Lustre: server umount lustre-MDT0000 complete [ 7352.877317] LustreError: 477691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771446698 with bad export cookie 8422920452594202951 [ 7352.880018] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7352.880419] LustreError: 477691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7352.992068] Lustre: server umount lustre-MDT0001 complete [ 7356.462167] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7357.405529] Key type lgssc unregistered [ 7357.524366] LNet: 479715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7357.527824] LNetError: 479715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7357.537318] LNet: Removed LNI 192.168.202.130@tcp [ 7357.820967] Key type .llcrypt unregistered [ 7357.822035] Key type ._llcrypt unregistered [ 7364.925576] Key type ._llcrypt registered [ 7364.926716] Key type .llcrypt registered [ 7364.963337] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7365.299370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7365.356265] alg: No test for adler32 (adler32-zlib) [ 7366.204599] Lustre: Lustre: Build Version: 2.17.50_152_g0212678 [ 7366.288908] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7367.871201] Key type lgssc registered [ 7368.258808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7380.424397] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 7383.864654] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7397.442128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7397.454198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.602031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7398.641401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7398.689875] Lustre: lustre-MDT0000: new disk, initializing [ 7398.735339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.750426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7400.076443] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7404.826519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.895760] Lustre: 484098:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7404.964203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7404.967604] Lustre: Skipped 1 previous similar message [ 7405.014479] Lustre: lustre-MDT0001: new disk, initializing [ 7405.059092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7405.074735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7405.080782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7406.701679] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7410.062731] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7411.546188] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7414.389092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7414.504891] Lustre: lustre-OST0000: new disk, initializing [ 7414.508200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7414.508406] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 7414.548054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.569427] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7419.584834] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7419.881799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7419.885317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7419.897355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7420.958417] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7421.022416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7422.411300] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7422.523851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7424.994797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7424.995183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7424.997182] LustreError: Skipped 1 previous similar message [ 7425.001570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7425.005727] Lustre: Skipped 1 previous similar message [ 7429.526552] Lustre: server umount lustre-OST0000 complete [ 7435.233624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.235238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7435.242043] Lustre: Skipped 2 previous similar messages [ 7437.017739] Lustre: server umount lustre-MDT0000 complete [ 7438.149136] LustreError: 484089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771446783 with bad export cookie 986455259235471502 [ 7438.152247] LustreError: 484089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7438.152337] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7438.257561] Lustre: server umount lustre-MDT0001 complete [ 7446.478562] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 7449.998793] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7462.164764] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7296 sec ========== 15:33:26 (1771446806) [ 7462.674041] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:33:26 (1771446806) === [ 7463.953212] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:33:27 (1771446807) === [ 7479.438314] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7480.866942] Key type lgssc unregistered [ 7481.022565] LNet: 492280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7481.027128] LNetError: 492280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7481.040604] LNet: Removed LNI 192.168.202.130@tcp [ 7481.526182] Key type .llcrypt unregistered [ 7481.528417] Key type ._llcrypt unregistered