[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474266788 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002009] Switched APIC routing to physical x2apic. [ 0.003009] kvm-guest: setup PV IPIs [ 0.005363] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007009] pid_max: default: 32768 minimum: 301 [ 0.008105] LSM: Security Framework initializing [ 0.009056] Yama: becoming mindful. [ 0.010042] SELinux: Initializing. [ 0.011073] *** VALIDATE selinux *** [ 0.018527] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022679] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023125] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025094] *** VALIDATE tmpfs *** [ 0.026340] *** VALIDATE proc *** [ 0.027159] *** VALIDATE cgroup *** [ 0.028006] *** VALIDATE cgroup2 *** [ 0.029192] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032033] Spectre V2 : User space: Vulnerable [ 0.033010] Speculative Store Bypass: Vulnerable [ 0.036558] debug: unmapping init [mem 0xffffffffad659000-0xffffffffad660fff] [ 0.039201] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041024] ... version: 2 [ 0.042014] ... bit width: 48 [ 0.043015] ... generic registers: 4 [ 0.044015] ... value mask: 0000ffffffffffff [ 0.045017] ... max period: 00007fffffffffff [ 0.046017] ... fixed-purpose events: 3 [ 0.047014] ... event mask: 000000070000000f [ 0.048328] rcu: Hierarchical SRCU implementation. [ 0.050592] smp: Bringing up secondary CPUs ... [ 0.051673] x86: Booting SMP configuration: [ 0.052032] .... node #0, CPUs: #1 #2 #3 [ 0.057098] smp: Brought up 1 node, 4 CPUs [ 0.059034] smpboot: Max logical packages: 1 [ 0.060024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141988] node 0 deferred pages initialised in 79ms [ 0.146151] devtmpfs: initialized [ 0.147229] x86/mm: Memory block size: 128MB [ 0.149757] gcov: version magic: 0x41383552 [ 0.150610] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156351] pinctrl core: initialized pinctrl subsystem [ 0.158169] [ 0.158707] ************************************************************* [ 0.161016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163012] ** ** [ 0.166016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168012] ** ** [ 0.171018] ** This means that this kernel is built to expose internal ** [ 0.173011] ** IOMMU data structures, which may compromise security on ** [ 0.176013] ** your system. ** [ 0.178014] ** ** [ 0.180013] ** If you see this message and you are not debugging the ** [ 0.182014] ** kernel, report this immediately to your vendor! ** [ 0.185013] ** ** [ 0.187011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ************************************************************* [ 0.191738] NET: Registered protocol family 16 [ 0.193431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200106] cpuidle: using governor menu [ 0.201690] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203526] PCI: Using configuration type 1 for base access [ 0.205125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213135] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.214036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216040] cryptd: max_cpu_qlen set to 1000 [ 0.218228] ACPI: Added _OSI(Module Device) [ 0.220015] ACPI: Added _OSI(Processor Device) [ 0.222010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228543] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235566] ACPI: Interpreter enabled [ 0.237067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238023] ACPI: Using IOAPIC for interrupt routing [ 0.240166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243529] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254611] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267621] acpiphp: Slot [2] registered [ 0.269226] acpiphp: Slot [5] registered [ 0.270226] acpiphp: Slot [6] registered [ 0.272132] acpiphp: Slot [7] registered [ 0.273194] acpiphp: Slot [8] registered [ 0.275128] acpiphp: Slot [9] registered [ 0.276187] acpiphp: Slot [10] registered [ 0.278122] acpiphp: Slot [3] registered [ 0.279104] acpiphp: Slot [4] registered [ 0.281112] acpiphp: Slot [11] registered [ 0.282096] acpiphp: Slot [12] registered [ 0.284123] acpiphp: Slot [13] registered [ 0.285099] acpiphp: Slot [14] registered [ 0.286257] acpiphp: Slot [15] registered [ 0.288133] acpiphp: Slot [16] registered [ 0.289166] acpiphp: Slot [17] registered [ 0.291121] acpiphp: Slot [18] registered [ 0.292107] acpiphp: Slot [19] registered [ 0.294134] acpiphp: Slot [20] registered [ 0.295114] acpiphp: Slot [21] registered [ 0.297129] acpiphp: Slot [22] registered [ 0.298111] acpiphp: Slot [23] registered [ 0.300101] acpiphp: Slot [24] registered [ 0.301114] acpiphp: Slot [25] registered [ 0.302139] acpiphp: Slot [26] registered [ 0.304140] acpiphp: Slot [27] registered [ 0.305150] acpiphp: Slot [28] registered [ 0.307132] acpiphp: Slot [29] registered [ 0.308252] acpiphp: Slot [30] registered [ 0.310150] acpiphp: Slot [31] registered [ 0.312097] PCI host bridge to bus 0000:00 [ 0.313021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.327224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.330199] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.333086] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.343025] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.353033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359160] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368998] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.388036] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395029] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402413] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419037] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457845] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.501594] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.514027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.521028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.579334] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.590021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.599023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.622020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.641364] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.649027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663032] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.701034] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.719104] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.745033] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.758033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.806021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831690] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.834452] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.837446] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.839550] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.842289] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.848248] iommu: Default domain type: Passthrough [ 0.849606] SCSI subsystem initialized [ 0.851181] ACPI: bus type USB registered [ 0.853157] usbcore: registered new interface driver usbfs [ 0.855129] usbcore: registered new interface driver hub [ 0.857087] usbcore: registered new device driver usb [ 0.858219] pps_core: LinuxPPS API ver. 1 registered [ 0.860014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.863066] PTP clock support registered [ 0.865073] EDAC MC: Ver: 3.0.0 [ 0.867150] PCI: Using ACPI for IRQ routing [ 0.870068] NetLabel: Initializing [ 0.871016] NetLabel: domain hash size = 128 [ 0.873017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.875114] NetLabel: unlabeled traffic allowed by default [ 0.877219] vgaarb: loaded [ 0.879337] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.881025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.886451] clocksource: Switched to clocksource kvm-clock [ 1.001362] VFS: Disk quotas dquot_6.6.0 [ 1.003136] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.005910] *** VALIDATE ramfs *** [ 1.007075] *** VALIDATE hugetlbfs *** [ 1.008528] pnp: PnP ACPI init [ 1.010891] pnp: PnP ACPI: found 6 devices [ 1.028043] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.031625] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.033948] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.036445] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.039289] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.042118] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.045422] NET: Registered protocol family 2 [ 1.048181] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.053464] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.057529] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.063517] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.067443] TCP: Hash tables configured (established 65536 bind 65536) [ 1.069982] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.073443] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.076655] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.079695] NET: Registered protocol family 1 [ 1.083059] RPC: Registered named UNIX socket transport module. [ 1.085507] RPC: Registered udp transport module. [ 1.087382] RPC: Registered tcp transport module. [ 1.089382] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.091782] NET: Registered protocol family 44 [ 1.093102] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.095759] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.098080] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.100628] PCI: CLS 0 bytes, default 64 [ 1.102325] Unpacking initramfs... [ 2.468414] debug: unmapping init [mem 0xffff9d483cc54000-0xffff9d483ffbffff] [ 2.471904] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.473776] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.476161] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.953318] Initialise system trusted keyrings [ 2.954497] Key type blacklist registered [ 2.955927] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.963336] zbud: loaded [ 2.965758] *** VALIDATE nfs *** [ 2.966716] *** VALIDATE nfs4 *** [ 2.969741] pstore: using deflate compression [ 2.973257] Platform Keyring initialized [ 3.075835] NET: Registered protocol family 38 [ 3.078307] Key type asymmetric registered [ 3.080047] Asymmetric key parser 'x509' registered [ 3.082141] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.085452] io scheduler mq-deadline registered [ 3.087440] io scheduler kyber registered [ 3.089347] io scheduler bfq registered [ 3.092986] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.096752] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.100222] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.103483] ACPI: Power Button [PWRF] [ 3.110127] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.117580] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.130133] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.138076] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.152351] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.178916] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.206855] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.213206] Non-volatile memory driver v1.3 [ 3.215073] Linux agpgart interface v0.103 [ 3.248290] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.250545] vda: detected capacity change from 0 to 74698752 [ 3.265025] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.267637] vdb: detected capacity change from 0 to 1073741824 [ 3.284332] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287454] vdc: detected capacity change from 0 to 2621440000 [ 3.310761] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.314583] vdd: detected capacity change from 0 to 2621440000 [ 3.336882] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.340064] vde: detected capacity change from 0 to 4294967296 [ 3.365984] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.370274] vdf: detected capacity change from 0 to 4294967296 [ 3.387214] libphy: Fixed MDIO Bus: probed [ 3.394737] usbcore: registered new interface driver usbserial_generic [ 3.397302] usbserial: USB Serial support registered for generic [ 3.399861] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.404568] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.406829] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.409657] mousedev: PS/2 mouse device common for all mice [ 3.413562] rtc_cmos 00:05: RTC can wake from S4 [ 3.416316] rtc_cmos 00:05: registered as rtc0 [ 3.418349] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.420547] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.420975] intel_pstate: CPU model not supported [ 3.427114] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.431761] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.434436] hid: raw HID events driver (C) Jiri Kosina [ 3.437068] usbcore: registered new interface driver usbhid [ 3.439161] usbhid: USB HID core driver [ 3.440863] drop_monitor: Initializing network drop monitor service [ 3.443352] Initializing XFRM netlink socket [ 3.445572] NET: Registered protocol family 10 [ 3.448933] Segment Routing with IPv6 [ 3.450164] NET: Registered protocol family 17 [ 3.452345] mpls_gso: MPLS GSO support [ 3.457185] RAS: Correctable Errors collector initialized. [ 3.459525] AVX version of gcm_enc/dec engaged. [ 3.461343] AES CTR mode by8 optimization enabled [ 3.559248] sched_clock: Marking stable (3559213589, 0)->(4489027245, -929813656) [ 3.563305] registered taskstats version 1 [ 3.565656] Loading compiled-in X.509 certificates [ 3.567935] zswap: loaded using pool lzo/zbud [ 3.596228] Key type big_key registered [ 3.611191] Key type encrypted registered [ 3.613273] ima: No TPM chip found, activating TPM-bypass! [ 3.615900] ima: Allocated hash algorithm: sha1 [ 3.618178] ima: No architecture policies found [ 3.620321] evm: Initialising EVM extended attributes: [ 3.622624] evm: security.selinux [ 3.624218] evm: security.ima [ 3.625627] evm: security.capability [ 3.627240] evm: HMAC attrs: 0x1 [ 3.629875] rtc_cmos 00:05: setting system clock to 2026-08-12 20:09:16 UTC (1786565356) [ 3.637144] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 3.640438] debug: unmapping init [mem 0xffffffffad382000-0xffffffffad658fff] [ 3.650099] Write protecting the kernel read-only data: 28672k [ 3.652948] debug: unmapping init [mem 0xffffffffaba03000-0xffffffffabbfffff] [ 3.655970] debug: unmapping init [mem 0xffffffffac314000-0xffffffffac3fffff] [ 3.690483] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.697575] systemd[1]: Detected virtualization kvm. [ 3.699045] systemd[1]: Detected architecture x86-64. [ 3.700103] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728944] systemd[1]: No hostname configured. [ 3.730085] systemd[1]: Set hostname to . [ 3.731437] random: systemd: uninitialized urandom read (16 bytes read) [ 3.733889] systemd[1]: Initializing machine ID from random generator. [ 3.768847] random: ln: uninitialized urandom read (6 bytes read) [ 3.869974] random: systemd: uninitialized urandom read (16 bytes read) [ 3.872784] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.877202] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.881831] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.516603] device-mapper: uevent: version 1.0.3 [ 4.518937] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 5.741082] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.229700] virtio_net virtio0 ens2: renamed from eth0 [ 6.866358] scsi host0: ata_piix [ 7.047523] scsi host1: ata_piix [ 7.049118] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.054620] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.161778] random: crng init done [ 11.171825] random: 7 urandom warning(s) missed due to ratelimiting [ 14.209360] dracut-initqueue[596]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ 15.964275] hrtimer: interrupt took 4898983 ns [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.085907] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.732231] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.603599] SELinux: Disabled at runtime. [ 20.772924] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.789911] systemd[1]: Detected virtualization kvm. [ 20.793775] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.613808] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.619560] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.632535] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.645358] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.653330] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.667921] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.686271] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required[ 23.190661] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 24.622638] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.926881] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.997075] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.502679] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.040989] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 33.195671] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 33.991589] NFS: Registering the id_resolver key type [ 33.996655] Key type id_resolver registered [ 34.005668] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg206-server login: [ 105.627236] libcfs: loading out-of-tree module taints kernel. [ 105.692270] Key type ._llcrypt registered [ 105.694421] Key type .llcrypt registered [ 105.789977] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 126.146381] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 128.199943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.251701] alg: No test for adler32 (adler32-zlib) [ 130.222274] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 131.835625] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 133.784311] Key type lgssc registered [ 136.307776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.096490] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 210.876412] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 225.726776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 225.766712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.065646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 227.117230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 227.225730] Lustre: lustre-MDT0000: new disk, initializing [ 227.356224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.403264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.636939] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.678321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.883321] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 250.939413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 250.943281] Lustre: Skipped 1 previous similar message [ 251.117949] Lustre: lustre-MDT0001: new disk, initializing [ 251.284945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.350123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.354171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 257.251186] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.735243] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 273.907660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.242599] Lustre: lustre-OST0000: new disk, initializing [ 274.249035] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 274.254889] Lustre: 8461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 274.298669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.602771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 279.622039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 279.766964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 281.614859] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 297.954878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 298.086717] Lustre: lustre-OST0001: new disk, initializing [ 298.089802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 298.094899] Lustre: 9533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 298.148584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 305.629775] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 306.228995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 306.236506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 306.327159] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 318.066290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 325.374431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 332.717339] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing check_logdir /tmp/testlogs/ [ 337.459800] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing yml_node [ 341.645093] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 344.700476] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 347.427787] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 349.132630] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 12 16:14:59 EDT 2026 [ 367.374286] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 369.320084] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 388.065546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.068857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.078218] Lustre: Skipped 2 previous similar messages [ 388.093258] Lustre: Skipped 3 previous similar messages [ 391.792586] Lustre: server umount lustre-MDT0000 complete [ 395.783064] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565748 with bad export cookie 11666079478909470926 [ 395.792078] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.305468] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.308116] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.334888] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 398.359162] Lustre: Skipped 1 previous similar message [ 398.367945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 398.378342] Lustre: Skipped 1 previous similar message [ 400.356256] LustreError: 11311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.356835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 400.395662] LustreError: 11311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 400.417211] Lustre: Skipped 1 previous similar message [ 402.314974] Lustre: server umount lustre-MDT0001 complete [ 406.450468] Lustre: server umount lustre-OST0000 complete [ 410.871941] Lustre: server umount lustre-OST0001 complete [ 418.271791] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 428.014653] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 466.109487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.364197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 466.395094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 466.473569] Lustre: lustre-MDT0000: new disk, initializing [ 466.596044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.623181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.722785] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 483.398257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.574664] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 483.657096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 483.660115] Lustre: Skipped 1 previous similar message [ 483.735550] Lustre: lustre-MDT0001: new disk, initializing [ 483.885967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 483.933255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 483.939199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 488.792646] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 498.060711] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 504.369923] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 511.787491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.327727] Lustre: lustre-OST0000: new disk, initializing [ 512.338164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 512.351823] Lustre: 17059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 512.540071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.438156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 514.446263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 514.595710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 520.972239] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 532.597831] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.525640] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 537.719870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 543.053637] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 543.297153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 545.254570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.264103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 545.269236] Lustre: Skipped 1 previous similar message [ 545.281125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.374065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.385389] Lustre: Skipped 2 previous similar messages [ 551.445692] Lustre: server umount lustre-OST0000 complete [ 560.611571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.614634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 560.645430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 560.659210] LustreError: Skipped 1 previous similar message [ 560.679755] Lustre: Skipped 1 previous similar message [ 570.336245] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 570.591854] Lustre: server umount lustre-MDT0000 complete [ 570.851547] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.866264] LustreError: 15098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 574.181725] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565927 with bad export cookie 11666079478909472459 [ 574.186952] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.190630] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 574.595122] Lustre: server umount lustre-MDT0001 complete [ 579.384685] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:18:49 (1786565929) [ 582.056458] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 583.927743] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:18:54 (1786565934) [ 591.432069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.959192] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.100298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.474572] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.509276] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 597.692608] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 606.691307] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.711845] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 607.772738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.837170] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 620.421367] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 625.777885] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 633.295228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 641.602242] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 650.705578] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 662.501967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 662.503498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.508437] LustreError: Skipped 1 previous similar message [ 662.510691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 662.519891] Lustre: Skipped 2 previous similar messages [ 662.540664] Lustre: Skipped 3 previous similar messages [ 664.602175] Lustre: server umount lustre-OST0000 complete [ 669.668216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 672.737815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.746074] Lustre: Skipped 1 previous similar message [ 675.332632] Lustre: server umount lustre-MDT0000 complete [ 677.857226] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.870767] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 679.326225] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566032 with bad export cookie 11666079478909473334 [ 679.339225] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.343572] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 679.638450] Lustre: server umount lustre-MDT0001 complete [ 690.047940] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 693.330430] Key type lgssc unregistered [ 693.628483] LNet: 23008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.631935] LNetError: 23008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.649962] LNet: Removed LNI 192.168.202.106@tcp [ 695.017135] Key type .llcrypt unregistered [ 695.018520] Key type ._llcrypt unregistered [ 719.860323] Key type ._llcrypt registered [ 719.861528] Key type .llcrypt registered [ 719.923380] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 720.962643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.070097] alg: No test for adler32 (adler32-zlib) [ 722.142867] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 722.396171] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 724.112232] Key type lgssc registered [ 725.185693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 733.237171] loop: module loaded [ 733.382726] loop0: detected capacity change from 0 to 204800000 [ 733.685729] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.697319] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 733.971070] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.100536] loop0: detected capacity change from 0 to 204800000 [ 740.372617] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.389409] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.408364] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 745.495980] loop0: detected capacity change from 0 to 204800000 [ 751.588759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.613217] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.910424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.961596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 753.105252] Lustre: lustre-MDT0000: new disk, initializing [ 753.231228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.253573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 758.486666] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 767.227875] loop1: detected capacity change from 0 to 204800000 [ 773.333270] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 773.727776] Lustre: lustre-OST07c6: new disk, initializing [ 773.733425] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 773.739123] Lustre: Skipped 1 previous similar message [ 773.743584] Lustre: 26193:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 773.836329] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 779.718492] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 782.481527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 782.488527] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 782.531570] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 785.060649] Lustre: 26709:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 785.083276] Lustre: 26709:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 785.094659] Lustre: 26709:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 785.109597] Lustre: 26709:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 785.119443] Lustre: 26709:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 785.124589] Lustre: 26709:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 800.850802] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 800.871654] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 800.877457] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 800.883698] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 800.889455] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 800.895430] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.615562] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 805.626022] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 805.632168] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 805.642724] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.648652] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.654271] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.661463] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 805.669029] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.675183] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 805.680897] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 805.687424] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 805.692903] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.843042] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 808.853954] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 808.860523] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 808.865693] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.875551] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.882812] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.888452] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 808.895570] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.903175] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 808.907957] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.915203] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.922878] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.651339] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 813.663869] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.673054] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 813.688244] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.698056] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.712475] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.722192] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 813.737777] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.742978] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 813.747888] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.754351] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.766526] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.162256] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 825.179992] Lustre: 25181:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 825.186103] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 825.195162] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 825.209434] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 825.224024] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 825.238183] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 825.249728] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 825.263867] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 825.275518] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 825.283142] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 825.289536] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 874.467967] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.483291] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 879.585111] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 884.709193] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 888.288352] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 888.489150] Lustre: server umount lustre-OST07c6 complete [ 897.078231] Lustre: server umount lustre-MDT0000 complete [ 913.997534] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 916.773598] Key type lgssc unregistered [ 917.089490] LNet: 28757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.104479] LNetError: 28757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.136237] LNet: Removed LNI 192.168.202.106@tcp [ 918.354524] Key type .llcrypt unregistered [ 918.357324] Key type ._llcrypt unregistered [ 936.240957] Key type ._llcrypt registered [ 936.247202] Key type .llcrypt registered [ 936.379604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 937.320780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 937.453625] alg: No test for adler32 (adler32-zlib) [ 938.615326] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 938.881873] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 940.568575] Key type lgssc registered [ 942.042934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.662902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.680746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.193094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.532390] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 961.510643] LustreError: 30028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.523876] LustreError: 30028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 962.801596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.236184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.726372] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 975.665261] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 981.598246] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 990.200837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.537234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.760055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 997.048155] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1005.301558] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1024.492154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.500544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1024.511447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.929306] Lustre: server umount lustre-OST0000 complete [ 1034.728589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.736948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1034.739759] Lustre: Skipped 1 previous similar message [ 1034.745388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.745402] Lustre: Skipped 1 previous similar message [ 1039.674259] Lustre: server umount lustre-MDT0000 complete [ 1039.844936] LustreError: 32811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.307076] LustreError: 32810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566397 with bad export cookie 11661531393916725866 [ 1044.312444] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.318153] LustreError: 32810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1044.920949] Lustre: server umount lustre-MDT0001 complete [ 1055.643814] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1058.334691] Key type lgssc unregistered [ 1058.639967] LNet: 33768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.649814] LNetError: 33768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.678816] LNet: Removed LNI 192.168.202.106@tcp [ 1059.277243] Key type .llcrypt unregistered [ 1059.279617] Key type ._llcrypt unregistered [ 1076.975819] Key type ._llcrypt registered [ 1076.977496] Key type .llcrypt registered [ 1077.085292] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1078.575427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.618280] alg: No test for adler32 (adler32-zlib) [ 1079.756892] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1079.990190] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1081.720248] Key type lgssc registered [ 1082.622154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1089.196429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.220088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.746266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.414555] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1100.266574] LustreError: 35036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.275766] LustreError: 35036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1103.483370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.798265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.810758] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1117.015758] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1122.877951] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1129.741422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1130.102589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.267196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1136.985738] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1142.504667] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1154.024742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.027933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.041746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.052554] Lustre: Skipped 1 previous similar message [ 1159.141565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.146489] Lustre: Skipped 1 previous similar message [ 1164.258283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.275129] Lustre: Skipped 1 previous similar message [ 1168.352275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1168.653171] Lustre: server umount lustre-OST0000 complete [ 1174.498147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.501527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1174.515894] Lustre: Skipped 2 previous similar messages [ 1174.541959] Lustre: Skipped 1 previous similar message [ 1178.634576] Lustre: server umount lustre-MDT0000 complete [ 1179.625963] LustreError: 35032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.662757] LustreError: 35032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1182.506872] LustreError: 35019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566535 with bad export cookie 6074931447498028432 [ 1182.515895] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1182.524050] LustreError: 35019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1182.832663] Lustre: server umount lustre-MDT0001 complete [ 1194.102384] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1196.533494] Key type lgssc unregistered [ 1196.743936] LNet: 38575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.749542] LNetError: 38575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.761275] LNet: Removed LNI 192.168.202.106@tcp [ 1197.346599] Key type .llcrypt unregistered [ 1197.348162] Key type ._llcrypt unregistered [ 1217.617375] Key type ._llcrypt registered [ 1217.619732] Key type .llcrypt registered [ 1217.717350] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1218.822266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1218.840510] alg: No test for adler32 (adler32-zlib) [ 1219.992124] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1220.212885] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1221.896202] Key type lgssc registered [ 1222.936710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.480588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.493994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.875840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.035784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1240.366465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.597057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.978213] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1250.066882] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1255.255372] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1260.776539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.049995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.935085] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1271.768580] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1276.402360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1281.505320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1281.508234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.510348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.530249] Lustre: Skipped 2 previous similar messages [ 1282.753570] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1282.759498] Lustre: Skipped 2 previous similar messages [ 1283.349446] Lustre: server umount lustre-MDT0000 complete [ 1286.637935] LustreError: 39968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.657018] LustreError: 39968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1286.773445] LustreError: 39954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566639 with bad export cookie 9512706948237938514 [ 1286.781622] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1286.793100] LustreError: 39954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1287.840946] LustreError: 41523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.860840] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [ 1291.745028] LustreError: 39967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.746503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.770436] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1293.170992] Lustre: server umount lustre-MDT0001 complete [ 1313.241968] Lustre: server umount lustre-OST0000 complete [ 1325.195523] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1327.471712] Key type lgssc unregistered [ 1327.679664] LNet: 43833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.686849] LNetError: 43833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.698477] LNet: Removed LNI 192.168.202.106@tcp [ 1328.253166] Key type .llcrypt unregistered [ 1328.254770] Key type ._llcrypt unregistered [ 1343.548543] Key type ._llcrypt registered [ 1343.551149] Key type .llcrypt registered [ 1343.647583] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1344.449404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.505449] alg: No test for adler32 (adler32-zlib) [ 1345.494103] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1345.669364] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1347.336712] Key type lgssc registered [ 1348.250582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1354.583925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1354.612494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.034844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.313135] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1365.478949] LustreError: 45227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.489921] LustreError: 45227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1366.357128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.669779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.896493] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1374.898370] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1379.063352] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1383.939842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1384.185902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.709875] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1392.383359] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1400.559284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1409.195240] Lustre: lustre-OST0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1411.043402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1411.043439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1411.050285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1411.061921] Lustre: Skipped 1 previous similar message [ 1411.069681] Lustre: Skipped 1 previous similar message [ 1413.545312] Lustre: server umount lustre-OST0000 complete [ 1431.394492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.402820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1431.403816] Lustre: Skipped 1 previous similar message [ 1434.528766] Lustre: server umount lustre-MDT0000 complete [ 1436.641760] LustreError: 45228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.648705] LustreError: 45228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1436.935765] LustreError: 45207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566789 with bad export cookie 7121646653745058550 [ 1436.939208] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.942201] LustreError: 45207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1443.228748] Lustre: server umount lustre-MDT0001 complete [ 1450.842157] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1452.615501] Key type lgssc unregistered [ 1452.853743] LNet: 48955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.864827] LNetError: 48955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.884751] LNet: Removed LNI 192.168.202.106@tcp [ 1453.426465] Key type .llcrypt unregistered [ 1453.428283] Key type ._llcrypt unregistered [ 1469.002969] Key type ._llcrypt registered [ 1469.005239] Key type .llcrypt registered [ 1469.072871] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1469.613879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.682337] alg: No test for adler32 (adler32-zlib) [ 1470.608406] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1470.711579] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1472.320184] Key type lgssc registered [ 1473.053557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.755605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.770407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.100247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.070822] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1487.809112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.085043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.889531] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1495.377891] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1498.933281] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.083680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1503.336900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.369168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1506.658506] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1510.330777] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1515.206896] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1517.079050] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1528.355122] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1216h34m38s [ 1529.269730] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1533.101620] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1533.411421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.422911] Lustre: Skipped 1 previous similar message [ 1534.176121] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1534.180668] Lustre: Skipped 3 previous similar messages [ 1538.549345] Lustre: server umount lustre-MDT0000 complete [ 1539.233318] LustreError: 50347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.242344] LustreError: 50347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1540.584151] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.600324] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1540.725335] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1544.935539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.019659] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1545.187089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.222982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1546.228314] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1547.788884] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1550.158108] Lustre: DEBUG MARKER: Wait for df (36765) ... [ 1551.317705] Lustre: DEBUG MARKER: done [ 1555.428106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1555.433082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.439905] Lustre: Skipped 1 previous similar message [ 1555.442984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.445419] Lustre: Skipped 3 previous similar messages [ 1556.451966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.207431] Lustre: server umount lustre-OST0000 complete [ 1566.697887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1566.700269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.724477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.729792] Lustre: Skipped 3 previous similar messages [ 1568.671954] Lustre: server umount lustre-MDT0000 complete [ 1571.106826] LustreError: 50332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566923 with bad export cookie 10486653661103986245 [ 1571.113403] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.114646] LustreError: 50332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1571.345126] Lustre: server umount lustre-MDT0001 complete [ 1577.034662] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1578.639556] Key type lgssc unregistered [ 1578.820645] LNet: 55278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.825491] LNetError: 55278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.837753] LNet: Removed LNI 192.168.202.106@tcp [ 1579.296574] Key type .llcrypt unregistered [ 1579.298018] Key type ._llcrypt unregistered [ 1600.416844] Key type ._llcrypt registered [ 1600.420178] Key type .llcrypt registered [ 1600.499129] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1601.277675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.294802] alg: No test for adler32 (adler32-zlib) [ 1602.218210] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1602.345762] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1603.968211] Key type lgssc registered [ 1604.696362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1610.003921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1610.019776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.216078] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1611.228528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1611.324107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.152249] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1624.559067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.661440] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1624.698795] Lustre: 57462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1624.870567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.490923] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1639.711252] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1646.987507] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1655.246085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1655.647690] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1655.981399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.333634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1665.632161] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1675.522636] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1682.939442] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1216h37m13s [ 1685.105696] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1692.128055] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1695.667711] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1695.748113] LustreError: 57471:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1715.901956] Lustre: lustre-MDT0000: Client f2438b24-ac57-4dcd-8f39-75219886daed (at 192.168.202.6@tcp) reconnecting [ 1735.768126] LustreError: 57471:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1737.848638] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1739.717305] Lustre: DEBUG MARKER: done [ 1746.404588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.408082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.429858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.453843] Lustre: Skipped 1 previous similar message [ 1750.593683] Lustre: server umount lustre-OST0000 complete [ 1756.643835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.654317] Lustre: Skipped 2 previous similar messages [ 1756.658948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1756.669812] Lustre: Skipped 1 previous similar message [ 1757.738544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.745924] Lustre: Skipped 1 previous similar message [ 1760.734853] Lustre: server umount lustre-MDT0000 complete [ 1764.075471] LustreError: 57455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567116 with bad export cookie 5644985784808007773 [ 1764.089359] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.587807] Lustre: server umount lustre-MDT0001 complete [ 1775.580779] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1778.218015] Key type lgssc unregistered [ 1778.454578] LNet: 61796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.462538] LNetError: 61796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.474596] LNet: Removed LNI 192.168.202.106@tcp [ 1779.221596] Key type .llcrypt unregistered [ 1779.223866] Key type ._llcrypt unregistered [ 1819.754935] Key type ._llcrypt registered [ 1819.756462] Key type .llcrypt registered [ 1819.867853] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1820.877551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.895356] alg: No test for adler32 (adler32-zlib) [ 1821.986857] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1822.232700] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1823.882368] Key type lgssc registered [ 1824.825766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.221106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.237566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.459835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1828.475886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1828.685410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.656502] Lustre: Failing over lustre-MDT0000 [ 1830.060577] Lustre: server umount lustre-MDT0000 complete [ 1837.228211] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:39:47 (1786567187) [ 1843.195700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.624817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.803736] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1856.083954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.157127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1856.192499] Lustre: 64570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1856.463860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.431308] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1867.540397] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1872.038341] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1877.421534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.572791] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1877.785534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.305317] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1885.913815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1890.186995] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1896.121742] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1902.561289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.568873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.589179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.659538] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.662091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.682637] Lustre: Skipped 2 previous similar messages [ 1907.957201] Lustre: server umount lustre-MDT0000 complete [ 1911.386429] LustreError: 64565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567264 with bad export cookie 2561718632176873345 [ 1911.388553] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.404668] LustreError: 64565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1911.652954] Lustre: server umount lustre-MDT0001 complete [ 1914.815499] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1919.443602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1927.140943] Lustre: 63496:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786567264/real 1786567264] req@0000000091c7de71 x1873351462132352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786567280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1927.178828] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.195369] Lustre: Skipped 1 previous similar message [ 1928.306317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.256148] Lustre: 63497:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786567269/real 1786567269] req@00000000e0051e2f x1873351462132480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786567285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1936.417035] LustreError: 63495:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a2389312 x1873351462133248/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.257287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.455680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1938.154488] Lustre: 63498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786567274/real 1786567274] req@00000000119e7f22 x1873351462132608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786567290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1942.757630] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1943.266991] Lustre: 63499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786567280/real 1786567280] req@00000000e3ca3004 x1873351462132992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786567296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1943.528182] LustreError: 68339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.555540] LustreError: 68339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1947.633424] LustreError: 68339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.648698] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1947.665135] LustreError: 68339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1951.441734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.958232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.306197] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1957.363516] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1963.979696] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1969.659842] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1975.865700] Lustre: server umount lustre-MDT0000 complete [ 1977.828161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1977.843097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.860279] Lustre: Skipped 2 previous similar messages [ 1977.868080] LustreError: 68854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.896097] LustreError: 68854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1979.439748] LustreError: 64563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567332 with bad export cookie 2561718632176875620 [ 1979.440714] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.452978] LustreError: 64563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1982.147634] Lustre: server umount lustre-MDT0001 complete [ 1989.940175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1999.073113] Lustre: 63499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786567335/real 1786567335] req@000000008da171f2 x1873351462159104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786567351 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1999.095503] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.440280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.668348] LustreError: 66422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.736933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.793489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2010.202559] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2012.129209] Lustre: 63498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786567348/real 1786567348] req@000000003f188f18 x1873351462159744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786567364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2012.146263] Lustre: 63498:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2014.183936] LustreError: 71023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.199855] LustreError: 71023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2017.260041] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2018.871622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.656459] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2024.438873] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2029.216770] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2034.583581] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.060665] Lustre: server umount lustre-MDT0000 complete [ 2044.845571] LustreError: 65308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567397 with bad export cookie 2561718632176876999 [ 2044.852295] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.861037] LustreError: 65308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2044.907576] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.916651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2044.919860] Lustre: Skipped 1 previous similar message [ 2044.930134] LustreError: 71022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.949535] LustreError: 71022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2045.303284] Lustre: server umount lustre-MDT0001 complete [ 2052.252314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2055.886572] Lustre: DEBUG MARKER: files compared the same [ 2058.187778] Lustre: server umount lustre-OST0000 complete [ 2074.187057] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2076.812678] Key type lgssc unregistered [ 2077.087872] LNet: 74298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.092697] LNetError: 74298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.106972] LNet: Removed LNI 192.168.202.106@tcp [ 2077.880684] Key type .llcrypt unregistered [ 2077.882937] Key type ._llcrypt unregistered [ 2102.858257] Key type ._llcrypt registered [ 2102.866969] Key type .llcrypt registered [ 2102.999649] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2103.871635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2103.894071] alg: No test for adler32 (adler32-zlib) [ 2104.921240] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2105.133864] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2106.792200] Key type lgssc registered [ 2107.872844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2116.233470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2116.249446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.637430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.885837] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2127.841061] LustreError: 75717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.869345] LustreError: 75717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2132.961149] LustreError: 75718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.499221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.006550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.640725] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2149.594455] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2155.752623] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2163.974541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2164.319379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.592783] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2176.878100] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2187.906296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2198.498148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2198.498987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.510699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2198.525443] Lustre: Skipped 1 previous similar message [ 2198.545622] Lustre: Skipped 1 previous similar message [ 2201.613920] Lustre: server umount lustre-OST0000 complete [ 2208.747270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.752122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.764662] Lustre: Skipped 1 previous similar message [ 2208.781360] Lustre: Skipped 1 previous similar message [ 2213.857405] LustreError: 78216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.893298] LustreError: 78216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2214.102043] Lustre: server umount lustre-MDT0000 complete [ 2218.141949] LustreError: 75697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567571 with bad export cookie 1556733136947000064 [ 2218.152667] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.426493] Lustre: server umount lustre-MDT0001 complete [ 2231.082405] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2234.260936] Key type lgssc unregistered [ 2234.604474] LNet: 79276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.623958] LNetError: 79276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.654511] LNet: Removed LNI 192.168.202.106@tcp [ 2235.651707] Key type .llcrypt unregistered [ 2235.655147] Key type ._llcrypt unregistered [ 2264.651880] Key type ._llcrypt registered [ 2264.657486] Key type .llcrypt registered [ 2264.779572] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2266.168845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.229166] alg: No test for adler32 (adler32-zlib) [ 2267.394372] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2267.640655] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2269.288327] Key type lgssc registered [ 2270.529241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2279.584093] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2279.614597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.136580] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2300.256669] LustreError: 80670:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [ 2327.078346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.985363] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2345.026901] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2353.334978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.610943] Lustre: *** cfs_fail_loc=706, val=0*** [ 2354.255369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.369976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2359.267451] LustreError: 80696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.304057] LustreError: 80696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2360.668746] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2361.327630] LustreError: 81757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.366263] LustreError: 81757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2366.433950] LustreError: 80695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.454065] LustreError: 80695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2369.481217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.844060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2374.698454] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2383.036610] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2390.277882] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2395.113373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.128762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.151469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.631590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2395.634794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.680134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.120739] Lustre: server umount lustre-OST0000 complete [ 2405.860672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.861791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.863897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.902389] Lustre: Skipped 1 previous similar message [ 2409.990448] Lustre: server umount lustre-MDT0000 complete [ 2410.984433] LustreError: 81761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.014325] LustreError: 81761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2413.956810] LustreError: 80677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567766 with bad export cookie 2819438418398056744 [ 2413.977557] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.990042] LustreError: 80677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2414.191799] Lustre: server umount lustre-MDT0001 complete [ 2426.491564] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2428.922831] Key type lgssc unregistered [ 2429.269775] LNet: 84288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.274610] LNetError: 84288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2430.310713] LNet: Removed LNI 192.168.202.106@tcp [ 2431.315151] Key type .llcrypt unregistered [ 2431.318708] Key type ._llcrypt unregistered [ 2455.625337] Key type ._llcrypt registered [ 2455.628831] Key type .llcrypt registered [ 2455.750932] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2457.026187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.042283] alg: No test for adler32 (adler32-zlib) [ 2458.076252] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2458.372896] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2460.104182] Key type lgssc registered [ 2461.324509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2468.454109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2468.466909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.904518] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2481.708693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.293614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2486.981443] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2487.783625] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.799468] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2492.903410] LustreError: 86395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.920811] LustreError: 86395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2494.736691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.017507] LustreError: 86392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.052796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.070951] LustreError: 86395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.090069] LustreError: 86395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2501.029635] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2505.186067] LustreError: 86396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.209092] LustreError: 86396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2509.078629] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.114191] Lustre: MGS: Client 43d11c8e-2fa4-4e97-a322-0cea0be4fd43 (at 0@lo) reconnecting [ 2509.153325] LustreError: 86406:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2509.439573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.509797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2513.866726] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2529.766835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.781794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.803494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2532.778720] Lustre: server umount lustre-OST0000 complete [ 2540.515655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.517657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2540.524822] Lustre: Skipped 1 previous similar message [ 2540.526741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.553729] Lustre: Skipped 3 previous similar messages [ 2542.492181] Lustre: server umount lustre-MDT0000 complete [ 2545.640195] LustreError: 86391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.673964] LustreError: 86391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2546.940638] Lustre: server umount lustre-MDT0001 complete [ 2551.162384] Lustre: server umount MGS complete [ 2563.405217] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:51:53 (1786567913) [ 2591.933988] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 2600.976781] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2636.344490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.874696] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2654.342803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.637452] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2654.702617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2654.922354] Lustre: lustre-MDT0001: new disk, initializing [ 2655.076864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2655.137110] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2655.151534] Lustre: 93565:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2655.166035] LustreError: 93565:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2660.812935] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2671.550903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.817822] Lustre: lustre-OST0000: new disk, initializing [ 2671.820746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2671.825506] Lustre: 94330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2671.907918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.499510] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2681.955192] Lustre: 94787:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000e2c9ef83 x1873352129028224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786568050 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2688.330348] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2695.886787] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.918299] Lustre: MGS: Client ca3ad709-9486-4fbe-82f8-f0a16c22831e (at 0@lo) reconnecting [ 2696.005899] Lustre: 92832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2696.016095] LustreError: 93551:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2696.060265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2696.195204] Lustre: lustre-MDT0000: new disk, initializing [ 2696.374098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.419948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2696.475865] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2697.508970] Lustre: 95400:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007f5f4557 x1873352129034880/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786568066 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2701.049536] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2702.690556] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2706.612207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2706.629680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2706.750691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2732.516114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.521473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.531690] Lustre: Skipped 1 previous similar message [ 2732.537143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.081102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.090311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2735.104205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.528428] Lustre: server umount lustre-OST0000 complete [ 2745.314718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.315167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2745.326576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.326585] Lustre: Skipped 1 previous similar message [ 2747.712307] Lustre: server umount lustre-MDT0000 complete [ 2750.446154] LustreError: 93536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.467735] LustreError: 93536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2758.097210] Lustre: server umount lustre-MDT0001 complete [ 2761.554890] Lustre: server umount MGS complete [ 2774.166864] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:55:24 (1786568124) [ 2790.149457] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2792.820564] Key type lgssc unregistered [ 2793.099060] LNet: 97898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2793.110432] LNetError: 97898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2793.124450] LNet: Removed LNI 192.168.202.106@tcp [ 2794.024420] Key type .llcrypt unregistered [ 2794.026427] Key type ._llcrypt unregistered [ 2811.876066] Key type ._llcrypt registered [ 2811.879979] Key type .llcrypt registered [ 2811.965617] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2813.120922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.151218] alg: No test for adler32 (adler32-zlib) [ 2814.436214] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2814.686487] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2816.497921] Key type lgssc registered [ 2817.555574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2824.626533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2824.637865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.085473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.849289] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2835.427347] LustreError: 99121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.450573] LustreError: 99121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2840.546867] LustreError: 99122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.260847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.592298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.190790] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2853.043646] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2858.704328] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2865.018458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2865.331971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.452644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2872.168435] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2879.315770] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2887.139552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2887.159208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.188404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.226627] Lustre: server umount lustre-OST0000 complete [ 2894.316328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.319183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.349111] Lustre: Skipped 1 previous similar message [ 2894.361932] Lustre: Skipped 1 previous similar message [ 2899.428397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.436266] Lustre: Skipped 1 previous similar message [ 2903.639144] Lustre: server umount lustre-MDT0000 complete [ 2904.556309] LustreError: 99122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.588846] LustreError: 99122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2907.197821] LustreError: 99100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568260 with bad export cookie 10386037365016528239 [ 2907.206146] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2907.209687] LustreError: 99100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2907.475311] Lustre: server umount lustre-MDT0001 complete [ 2917.610123] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2920.707904] Key type lgssc unregistered [ 2921.078603] LNet: 102658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.083144] LNetError: 102658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.098944] LNet: Removed LNI 192.168.202.106@tcp [ 2921.853562] Key type .llcrypt unregistered [ 2921.858772] Key type ._llcrypt unregistered [ 2937.609819] Key type ._llcrypt registered [ 2937.617311] Key type .llcrypt registered [ 2937.712093] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2938.872607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2938.938464] alg: No test for adler32 (adler32-zlib) [ 2940.029485] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2940.223105] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2941.907177] Key type lgssc registered [ 2942.959640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2948.071558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2948.074969] LustreError: 103742:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2948.076227] Lustre: Skipped 1 previous similar message [ 2948.076313] LustreError: 103741:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2948.091111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.091652] LustreError: 103742:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2948.103086] LustreError: 103742:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2948.108039] LustreError: 103742:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2949.647379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.402816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.624532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2960.355517] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2964.508911] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing lsmod [ 2966.923085] LustreError: 104948:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2966.923375] LustreError: 104947:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2966.940518] LustreError: 104948:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2966.952309] LustreError: 104948:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2966.972256] LustreError: 104948:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2966.990426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.310044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.129427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2972.140313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.159306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.644568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.659265] Lustre: Skipped 1 previous similar message [ 2972.669633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.685462] Lustre: Skipped 1 previous similar message [ 2977.762340] LustreError: 103780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.782589] LustreError: 103780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2977.937706] Lustre: server umount lustre-MDT0000 complete [ 2981.336563] LustreError: 105134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568334 with bad export cookie 7348827814468148273 [ 2981.340202] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.349947] LustreError: 105134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2981.632835] Lustre: server umount lustre-MDT0001 complete [ 2996.067207] Lustre: server umount lustre-OST0000 complete [ 3003.299057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.843489] LustreError: 106011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.961803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.248306] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3008.996164] LustreError: 106012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.116249] LustreError: 106011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.054874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.458487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3021.203910] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3026.783958] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3030.907741] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3036.160499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3036.644203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.784380] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3048.294467] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3048.961243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3059.173178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.187536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3059.191521] Lustre: Skipped 1 previous similar message [ 3059.210749] Lustre: Skipped 1 previous similar message [ 3064.293426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.300809] Lustre: Skipped 1 previous similar message [ 3064.699412] Lustre: server umount lustre-OST0000 complete [ 3068.902577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3068.908109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.918447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3073.839271] Lustre: server umount lustre-MDT0000 complete [ 3074.532209] LustreError: 106012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.541484] LustreError: 106012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3076.337685] LustreError: 108790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568429 with bad export cookie 7348827814468148987 [ 3076.339517] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3076.344300] LustreError: 108790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3076.582097] Lustre: server umount lustre-MDT0001 complete [ 3083.496949] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3085.411273] Key type lgssc unregistered [ 3085.643897] LNet: 109545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3085.649230] LNetError: 109545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3085.656833] LNet: Removed LNI 192.168.202.106@tcp [ 3086.353110] Key type .llcrypt unregistered [ 3086.362625] Key type ._llcrypt unregistered [ 3103.268463] Key type ._llcrypt registered [ 3103.270159] Key type .llcrypt registered [ 3103.342333] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3104.172411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3104.254516] alg: No test for adler32 (adler32-zlib) [ 3105.195301] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 3105.338429] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3106.976416] Key type lgssc registered [ 3107.878721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3113.442744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3113.464443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.795258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.829737] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3124.113847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.195194] LustreError: 110952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.210136] LustreError: 110952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3124.455240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.728807] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3133.434290] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3137.274710] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3142.616828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.854449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.623619] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3152.805954] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3159.284615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3166.299905] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3167.987287] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3170.788537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3170.795328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.805093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.886243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.897302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.897328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.921277] Lustre: Skipped 1 previous similar message [ 3176.310397] Lustre: server umount lustre-OST0000 complete [ 3180.002492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.012474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.019291] Lustre: Skipped 1 previous similar message [ 3183.018161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3183.028524] Lustre: Skipped 1 previous similar message [ 3185.494762] Lustre: server umount lustre-MDT0000 complete [ 3188.375387] LustreError: 110932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568541 with bad export cookie 4103252149862517094 [ 3188.380639] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3188.384481] LustreError: 110932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3188.585161] Lustre: server umount lustre-MDT0001 complete [ 3195.545159] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3197.466923] Key type lgssc unregistered [ 3197.648765] LNet: 114608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3197.657250] LNetError: 114608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3197.671388] LNet: Removed LNI 192.168.202.106@tcp [ 3198.374556] Key type .llcrypt unregistered [ 3198.378154] Key type ._llcrypt unregistered [ 3210.721350] Key type ._llcrypt registered [ 3210.722905] Key type .llcrypt registered [ 3210.787977] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3211.442773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3211.517565] alg: No test for adler32 (adler32-zlib) [ 3212.583890] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 3212.797585] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3214.466404] Key type lgssc registered [ 3215.367468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3221.342301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3221.366765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.762157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.926681] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3232.237633] LustreError: 115813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.250926] LustreError: 115813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3232.283411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.452402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3235.065977] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3240.080759] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3243.587226] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3247.960242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.236727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.609286] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3257.677252] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3267.714319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3278.305816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.311291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.324980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.331445] Lustre: Skipped 1 previous similar message [ 3280.802876] Lustre: server umount lustre-OST0000 complete [ 3288.549440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.555279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.561141] Lustre: Skipped 2 previous similar messages [ 3288.579128] Lustre: Skipped 1 previous similar message [ 3293.667674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.684839] Lustre: Skipped 1 previous similar message [ 3298.272364] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3298.643463] Lustre: server umount lustre-MDT0000 complete [ 3298.821772] LustreError: 115813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.837138] LustreError: 115813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3302.004245] LustreError: 115794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568654 with bad export cookie 5038315929881784516 [ 3302.010434] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3302.013856] LustreError: 115794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3302.291104] Lustre: server umount lustre-MDT0001 complete [ 3312.223554] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3314.645117] Key type lgssc unregistered [ 3314.889184] LNet: 119370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3314.895625] LNetError: 119370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3314.910284] LNet: Removed LNI 192.168.202.106@tcp [ 3316.048619] Key type .llcrypt unregistered [ 3316.051987] Key type ._llcrypt unregistered [ 3334.585654] Key type ._llcrypt registered [ 3334.587631] Key type .llcrypt registered [ 3334.659504] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3335.711584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3335.770427] alg: No test for adler32 (adler32-zlib) [ 3336.786802] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 3336.961128] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3338.608222] Key type lgssc registered [ 3339.578708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3345.641860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3345.656382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.044616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.721872] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3356.648210] LustreError: 120765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.657264] LustreError: 120765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3358.393545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.782696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3362.633124] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3369.120299] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3373.385481] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3379.092368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3379.419821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.304293] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3389.725365] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3391.742275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3400.891446] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3403.889300] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3403.892294] Lustre: Skipped 2 previous similar messages [ 3407.366394] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3407.370191] Lustre: Skipped 2 previous similar messages [ 3410.486663] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3410.503116] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3415.423823] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3415.434853] Lustre: Skipped 2 previous similar messages [ 3420.747514] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.6@tcp 192.168.0.[2,10]@tcp [ 3423.911522] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.6/24@tcp in log lustre-MDT0000 [ 3423.918129] Lustre: Skipped 2 previous similar messages [ 3429.912471] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.6/24@tcp [ 3429.919851] Lustre: Skipped 1 previous similar message [ 3437.037126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3437.042994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3437.045688] LustreError: Skipped 1 previous similar message [ 3437.061365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3438.055357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3438.064108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.296138] Lustre: server umount lustre-OST0000 complete [ 3445.735693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3445.748991] LustreError: Skipped 3 previous similar messages [ 3445.760031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.774544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3445.786519] Lustre: Skipped 1 previous similar message [ 3448.295409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.296622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.317728] Lustre: Skipped 1 previous similar message [ 3450.457300] Lustre: server umount lustre-MDT0000 complete [ 3452.925533] LustreError: 124088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568805 with bad export cookie 896306814363244306 [ 3452.926356] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.932260] LustreError: 124088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3453.177956] Lustre: server umount lustre-MDT0001 complete [ 3460.770694] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3462.915620] Key type lgssc unregistered [ 3463.185822] LNet: 125744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3463.197228] LNetError: 125744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3463.208765] LNet: Removed LNI 192.168.202.106@tcp [ 3463.792774] Key type .llcrypt unregistered [ 3463.795177] Key type ._llcrypt unregistered [ 3483.595373] Key type ._llcrypt registered [ 3483.597699] Key type .llcrypt registered [ 3483.744732] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3484.553674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3484.597300] alg: No test for adler32 (adler32-zlib) [ 3485.647374] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 3485.867781] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3487.523046] Key type lgssc registered [ 3488.604824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3494.629835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3494.646274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.023157] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3496.046463] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.6/24@tcp [ 3496.050260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.701953] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3505.646775] LustreError: 127283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.658379] LustreError: 127283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3506.052973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.271203] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3506.301761] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.6/24@tcp [ 3506.363249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.790062] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3515.652378] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3519.734796] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3524.399413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3524.669565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3528.803524] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3534.101828] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3541.036549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3551.204618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.211548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.220285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.228831] Lustre: Skipped 1 previous similar message [ 3556.325978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3556.330940] Lustre: Skipped 1 previous similar message [ 3557.341806] Lustre: server umount lustre-OST0000 complete [ 3561.443637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.451024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.454849] Lustre: Skipped 2 previous similar messages [ 3566.565167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.570169] Lustre: Skipped 2 previous similar messages [ 3566.705478] Lustre: server umount lustre-MDT0000 complete [ 3569.324469] LustreError: 127263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786568922 with bad export cookie 13933512674151138002 [ 3569.330174] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3569.334112] LustreError: 127263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3569.509857] Lustre: server umount lustre-MDT0001 complete [ 3576.334296] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3578.051860] Key type lgssc unregistered [ 3578.250549] LNet: 130896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3578.259365] LNetError: 130896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3578.279934] LNet: Removed LNI 192.168.202.106@tcp [ 3578.844691] Key type .llcrypt unregistered [ 3578.847780] Key type ._llcrypt unregistered [ 3597.922294] Key type ._llcrypt registered [ 3597.923712] Key type .llcrypt registered [ 3597.997034] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 3607.372437] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3607.946818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.016068] alg: No test for adler32 (adler32-zlib) [ 3608.977071] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 3609.139927] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3610.784727] Key type lgssc registered [ 3611.588780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.889510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3635.919398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.100376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3637.129169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3637.179546] Lustre: lustre-MDT0000: new disk, initializing [ 3637.227444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.240539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3640.007754] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3647.715701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.797371] Lustre: 134272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3647.846643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3647.849947] Lustre: Skipped 1 previous similar message [ 3647.896644] Lustre: lustre-MDT0001: new disk, initializing [ 3647.967554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.010252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3648.022472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3650.976175] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3658.242057] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3661.831305] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3667.018676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.269441] Lustre: lustre-OST0000: new disk, initializing [ 3667.272692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3667.277593] Lustre: 136239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3667.384294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.312731] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3674.158044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3674.172878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3674.230107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3678.925571] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3681.899032] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3682.014237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3685.201172] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3685.385767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3689.955912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3689.957805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.963019] LustreError: Skipped 1 previous similar message [ 3689.964959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3689.971340] Lustre: Skipped 1 previous similar message [ 3692.744288] Lustre: server umount lustre-OST0000 complete [ 3699.171983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3699.177099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.186029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.188097] Lustre: Skipped 1 previous similar message [ 3700.194548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.194715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.220098] Lustre: Skipped 1 previous similar message [ 3701.458409] Lustre: server umount lustre-MDT0000 complete [ 3704.164927] LustreError: 135074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569057 with bad export cookie 2279874331817021447 [ 3704.170117] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3704.173479] LustreError: 135074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3704.414725] Lustre: server umount lustre-MDT0001 complete [ 3711.032438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.378913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.890414] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3716.579281] LustreError: 138452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.589930] LustreError: 138452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3721.273217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.488984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.822037] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3730.505425] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3734.990583] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3740.292967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.574410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.335203] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3750.316926] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3753.395826] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3758.800685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3762.106748] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3762.278906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3768.989748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.109456] Lustre: lustre-OST0001: new disk, initializing [ 3769.111320] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3769.121492] Lustre: 141854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3769.172293] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3773.290909] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3778.098846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3778.109478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3778.199986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3778.825531] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3778.986795] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3781.535516] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3781.660285] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3785.249392] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d98480e7800.ost_server_uuid 50 [ 3786.211444] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d98480e7800.ost_server_uuid in FULL state after 0 sec [ 3793.897572] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3793.900597] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.907393] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3793.911209] Lustre: Skipped 1 previous similar message [ 3797.017583] Lustre: server umount lustre-OST0001 complete [ 3799.016141] LustreError: 140272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3799.034249] LustreError: 140272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3803.109412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.112206] LustreError: Skipped 1 previous similar message [ 3803.113492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.137739] Lustre: Skipped 1 previous similar message [ 3803.141184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.134163] LustreError: 142545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.144079] LustreError: 142545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3805.482269] Lustre: server umount lustre-OST0000 complete [ 3808.739170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3808.744629] LustreError: Skipped 1 previous similar message [ 3813.934710] Lustre: server umount lustre-MDT0000 complete [ 3814.373610] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.403785] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3816.596863] LustreError: 138434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569169 with bad export cookie 2279874331817022329 [ 3816.604810] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3816.610062] LustreError: 138434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3816.893250] Lustre: server umount lustre-MDT0001 complete [ 3829.325197] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3831.284869] Key type lgssc unregistered [ 3831.471847] LNet: 144856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3831.475536] LNetError: 144856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3831.488776] LNet: Removed LNI 192.168.202.106@tcp [ 3832.139801] Key type .llcrypt unregistered [ 3832.141619] Key type ._llcrypt unregistered [ 3858.250219] Key type ._llcrypt registered [ 3858.256865] Key type .llcrypt registered [ 3858.316178] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 3867.028914] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3867.656156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3867.666941] alg: No test for adler32 (adler32-zlib) [ 3868.576896] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 3868.728601] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3870.352208] Key type lgssc registered [ 3870.981798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3891.802736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3891.818717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.009592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3893.030794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3893.087553] Lustre: lustre-MDT0000: new disk, initializing [ 3893.141369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.160336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3895.458803] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3903.130521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3903.215463] Lustre: 148990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3903.250227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3903.252798] Lustre: Skipped 1 previous similar message [ 3903.303985] Lustre: lustre-MDT0001: new disk, initializing [ 3903.387659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3903.428124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3903.447990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3906.133174] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3912.702525] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3916.285429] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3920.891918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3921.074177] Lustre: lustre-OST0000: new disk, initializing [ 3921.076842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3921.081316] Lustre: 150955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3921.157451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.129843] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3930.670354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3930.680702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3930.759809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3931.367869] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3934.256187] Lustre: Failing over lustre-OST0000 [ 3934.376019] Lustre: server umount lustre-OST0000 complete [ 3950.764316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.042366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.069477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.083475] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3953.068193] Lustre: 147213:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786569289/real 1786569289] req@00000000fbf12cc3 x1873353607509376/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786569305 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3953.350412] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3955.269233] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3956.225434] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 3956.226798] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3957.282858] Lustre: 147214:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786569294/real 1786569294] req@000000006c808378 x1873353607510400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786569310 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3957.306019] Lustre: 147214:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3958.151298] Lustre: Failing over lustre-MDT0000 [ 3958.291657] Lustre: server umount lustre-MDT0000 complete [ 3959.778977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.783412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.790982] Lustre: Skipped 1 previous similar message [ 3961.317102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.323160] Lustre: Skipped 1 previous similar message [ 3962.336214] Lustre: 147215:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786569299/real 1786569299] req@00000000ebe70d80 x1873353607511552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786569315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3963.550871] LustreError: 148998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.565213] LustreError: 148998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3966.437379] LustreError: 149001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.463166] LustreError: 149001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3968.676653] LustreError: 148998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.552954] LustreError: 149002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.568460] LustreError: 149002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3974.332758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.407914] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3974.578378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.613307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3974.790021] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3977.514802] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3979.765176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3979.774724] Lustre: Skipped 1 previous similar message [ 3979.793869] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3979.817890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3984.866500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.871759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.872680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.891816] Lustre: Skipped 1 previous similar message [ 3989.109134] Lustre: server umount lustre-OST0000 complete [ 3995.105321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3995.110473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.113567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3995.125477] Lustre: Skipped 2 previous similar messages [ 3995.131889] Lustre: Skipped 1 previous similar message [ 3997.934462] Lustre: server umount lustre-MDT0000 complete [ 4000.225776] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.238294] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4000.680597] LustreError: 148983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569353 with bad export cookie 923544784651696915 [ 4000.686921] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.692108] LustreError: 148983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4000.926723] Lustre: server umount lustre-MDT0001 complete [ 4008.524046] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4010.310672] Key type lgssc unregistered [ 4010.526297] LNet: 154608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4010.530726] LNetError: 154608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4010.547348] LNet: Removed LNI 192.168.202.106@tcp [ 4011.143667] Key type .llcrypt unregistered [ 4011.147906] Key type ._llcrypt unregistered [ 4038.923377] Key type ._llcrypt registered [ 4038.925486] Key type .llcrypt registered [ 4038.990511] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4039.842238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4039.856762] alg: No test for adler32 (adler32-zlib) [ 4040.841906] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 4041.009301] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4042.608553] Key type lgssc registered [ 4043.390991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4048.559943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4048.571461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.821085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.521222] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4058.965602] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4060.130319] LustreError: 156130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.148932] LustreError: 156130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4062.062609] LustreError: 156158:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4062.067611] LustreError: 156158:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4062.074987] LustreError: 156158:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 4062.301106] Lustre: server umount lustre-MDT0000 complete [ 4072.645171] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4077.942382] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4099.369271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.515307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4099.531918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4099.573883] Lustre: lustre-MDT0000: new disk, initializing [ 4099.615861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4099.626050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4102.255985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4109.337660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4109.423535] Lustre: 159457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4109.459168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4109.461777] Lustre: Skipped 1 previous similar message [ 4109.514120] Lustre: lustre-MDT0001: new disk, initializing [ 4109.580517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4109.608244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4109.633494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4112.260737] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4118.237191] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4121.908361] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4126.048797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.181738] Lustre: lustre-OST0000: new disk, initializing [ 4126.184400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4126.188483] Lustre: 161424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4126.225982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.273236] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4132.372458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4132.388956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4132.482602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4135.767975] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4139.292824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4140.233101] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:18:11 (1786569491) [ 4283.872878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.872887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4283.873208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.877927] Lustre: Skipped 2 previous similar messages [ 4285.855770] Lustre: server umount lustre-MDT0000 complete [ 4287.890739] LustreError: 161423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569640 with bad export cookie 10995785682118443751 [ 4287.902346] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4288.998163] LustreError: 160371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.007079] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.025506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4289.030138] Lustre: Skipped 2 previous similar messages [ 4293.031341] LustreError: 159463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.031662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4298.145344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4298.145344] LustreError: 160371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.304123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4302.461936] Lustre: server umount lustre-MDT0001 complete [ 4319.200147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4319.339910] Lustre: server umount lustre-OST0000 complete [ 4326.031298] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4332.317716] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4354.555409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.681195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4354.706159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4354.761844] Lustre: lustre-MDT0000: new disk, initializing [ 4354.805485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.815947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4356.899505] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4363.637672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.697609] Lustre: 166068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4363.726220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4363.728159] Lustre: Skipped 1 previous similar message [ 4363.771253] Lustre: lustre-MDT0001: new disk, initializing [ 4363.839130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4363.873736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4363.883321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4366.684199] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4373.117321] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4376.570825] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4380.678058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4380.854365] Lustre: lustre-OST0000: new disk, initializing [ 4380.857400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4380.865441] Lustre: 168030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4380.911641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.040071] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4388.403513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4388.416960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4388.476517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4391.192288] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4393.661803] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4393.754763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.477669] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4396.589938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4398.564461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4398.568183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.574413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.585401] Lustre: Skipped 1 previous similar message [ 4398.590219] Lustre: Skipped 1 previous similar message [ 4403.775979] Lustre: server umount lustre-OST0000 complete [ 4408.806626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.820471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.826581] Lustre: Skipped 3 previous similar messages [ 4412.274428] Lustre: server umount lustre-MDT0000 complete [ 4413.921529] LustreError: 166077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.927876] LustreError: 166077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4414.637834] LustreError: 166060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569767 with bad export cookie 10995785682118602574 [ 4414.644808] LustreError: 166060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4414.648907] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.895696] Lustre: server umount lustre-MDT0001 complete [ 4419.768723] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:22:50 (1786569770) [ 4428.897324] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4433.386815] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4450.219108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.325910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4450.340105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4450.383799] Lustre: lustre-MDT0000: new disk, initializing [ 4450.415194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.428319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4452.364804] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4458.194880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.247840] Lustre: 172778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4458.287661] LustreError: 172789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.410247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4460.375775] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4465.480764] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4468.197663] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4471.559985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.760237] Lustre: 174753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4475.327476] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4479.510232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4479.592212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4480.928969] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4485.100198] Lustre: server umount lustre-OST0000 complete [ 4489.186093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4489.191223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.196334] Lustre: Skipped 1 previous similar message [ 4489.199209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4497.944895] Lustre: server umount lustre-MDT0000 complete [ 4499.938478] LustreError: 172789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.947143] LustreError: 172789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4500.165300] LustreError: 172770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569853 with bad export cookie 10995785682118603463 [ 4500.168525] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.173634] LustreError: 172770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4510.795316] Lustre: server umount lustre-MDT0001 complete [ 4515.719947] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:24:26 (1786569866) [ 4525.098108] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4531.423844] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4552.756514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.027341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4553.046292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4553.049757] Lustre: Skipped 3 previous similar messages [ 4553.123896] Lustre: lustre-MDT0000: new disk, initializing [ 4553.126937] Lustre: Skipped 2 previous similar messages [ 4553.214830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4553.222320] Lustre: Skipped 2 previous similar messages [ 4553.238824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4553.246214] Lustre: Skipped 2 previous similar messages [ 4556.020767] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4563.146574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.235538] Lustre: 179068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4563.433707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4566.267320] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4572.423802] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4576.194348] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4580.210884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.398858] Lustre: 181044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4584.308220] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4590.710131] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4590.721235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4595.683284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.686226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.689704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.689710] Lustre: Skipped 3 previous similar messages [ 4595.706675] Lustre: Skipped 2 previous similar messages [ 4597.816947] Lustre: server umount lustre-OST0000 complete [ 4606.634968] Lustre: server umount lustre-MDT0000 complete [ 4608.750216] LustreError: 179058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569961 with bad export cookie 10995785682118604548 [ 4608.756955] LustreError: 179058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4608.759253] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4609.022500] Lustre: server umount lustre-MDT0001 complete [ 4615.758803] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4617.672814] Key type lgssc unregistered [ 4617.868159] LNet: 182903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4617.873987] LNetError: 182903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4617.900743] LNet: Removed LNI 192.168.202.106@tcp [ 4618.576977] Key type .llcrypt unregistered [ 4618.579824] Key type ._llcrypt unregistered [ 4634.078174] Key type ._llcrypt registered [ 4634.079647] Key type .llcrypt registered [ 4634.140286] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4634.724438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4634.795489] alg: No test for adler32 (adler32-zlib) [ 4635.733933] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 4635.933924] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4637.576185] Key type lgssc registered [ 4638.218261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4643.167761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4643.189862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.465171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.416707] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4653.753342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.059674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4657.266457] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4662.791320] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4666.603018] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4671.211219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.394508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.496626] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4682.125785] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4689.878624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4709.345809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4709.354359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.366654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4710.373587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4710.374936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4710.384233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.910455] Lustre: server umount lustre-OST0000 complete [ 4715.490796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.508200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4715.516403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4715.521283] Lustre: Skipped 1 previous similar message [ 4720.609248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.611769] Lustre: Skipped 2 previous similar messages [ 4721.430925] Lustre: server umount lustre-MDT0000 complete [ 4723.730790] LustreError: 184280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570076 with bad export cookie 4692130267678348303 [ 4723.739573] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4723.927369] Lustre: server umount lustre-MDT0001 complete [ 4731.012671] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4732.797334] Key type lgssc unregistered [ 4733.014568] LNet: 187857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4733.017716] LNetError: 187857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4733.031077] LNet: Removed LNI 192.168.202.106@tcp [ 4733.501821] Key type .llcrypt unregistered [ 4733.503123] Key type ._llcrypt unregistered [ 4750.265719] Key type ._llcrypt registered [ 4750.267192] Key type .llcrypt registered [ 4750.362726] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4750.997929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4751.121597] alg: No test for adler32 (adler32-zlib) [ 4752.121355] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 4752.301555] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4753.936225] Key type lgssc registered [ 4754.582533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4759.654205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4759.678202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.977709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.583320] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4768.426759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.733716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4771.415421] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4775.732766] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4778.834933] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4782.760434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4782.965818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.032659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4786.146274] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4790.696337] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4801.505662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4801.509316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.521760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4803.256991] Lustre: lustre-OST0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 4803.554520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4803.566062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4805.628931] Lustre: server umount lustre-OST0000 complete [ 4810.136945] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d9843006800.ost_server_uuid 50 [ 4811.116060] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4821.835815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.848861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4821.853400] Lustre: Skipped 3 previous similar messages [ 4823.909316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4823.916028] Lustre: Skipped 1 previous similar message [ 4828.027645] Lustre: server umount lustre-MDT0000 complete [ 4829.154582] LustreError: 189252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.164548] LustreError: 189252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4830.267659] LustreError: 191827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570183 with bad export cookie 5924424312662156439 [ 4830.271604] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4830.279686] LustreError: 191827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4830.454776] Lustre: server umount lustre-MDT0001 complete [ 4835.704045] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:29:46 (1786570186) [ 4839.187572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4839.472131] LustreError: 193139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.520283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.984553] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4844.527401] LustreError: 193140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.449695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.695798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4850.064171] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4854.260212] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.680216] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4862.188912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4862.400711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.514116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4866.190553] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4870.608086] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4875.317107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4875.443219] Lustre: lustre-OST0001: new disk, initializing [ 4875.446634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4875.451278] Lustre: 195985:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4880.019199] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4885.559101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4885.572936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4885.628513] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4886.584719] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4890.081609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4890.086457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.114136] Lustre: Skipped 1 previous similar message [ 4890.121418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4894.888777] Lustre: server umount lustre-OST0000 complete [ 4896.233505] LustreError: 194971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.243914] LustreError: 194971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4899.251645] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4899.388958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4901.353192] LustreError: 194970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.362144] LustreError: 194970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4902.253434] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4902.382891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4911.592881] LustreError: 194971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.604465] LustreError: 194971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4916.707362] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4916.709941] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.715101] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4916.715112] Lustre: Skipped 3 previous similar messages [ 4916.715399] LustreError: Skipped 1 previous similar message [ 4916.720662] Lustre: Skipped 2 previous similar messages [ 4920.180338] Lustre: server umount lustre-OST0001 complete [ 4924.385512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4926.949936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.958106] Lustre: Skipped 1 previous similar message [ 4928.765894] Lustre: server umount lustre-MDT0000 complete [ 4931.193880] LustreError: 193121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570284 with bad export cookie 5924424312662157510 [ 4931.196056] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4931.202742] LustreError: 193121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4931.438546] Lustre: server umount lustre-MDT0001 complete [ 4945.032700] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:31:36 (1786570296) [ 4949.030268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.235837] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4949.278252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4949.396828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.399567] Lustre: Skipped 1 previous similar message [ 4951.771499] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4956.784232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.831446] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4956.849803] Lustre: 199423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.257941] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4963.365857] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4966.409343] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4970.410210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4970.519767] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4973.809117] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4977.880473] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4978.210481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4981.610908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4981.649417] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4981.757725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4981.760554] Lustre: Skipped 2 previous similar messages [ 4984.881519] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4986.453823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4988.963930] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4991.463495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.469694] Lustre: Skipped 1 previous similar message [ 4991.469803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4991.479581] Lustre: Skipped 5 previous similar messages [ 4996.441977] Lustre: server umount lustre-OST0000 complete [ 4996.581806] LustreError: 201272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.588770] LustreError: 201272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5011.937976] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5011.942183] LustreError: Skipped 1 previous similar message [ 5014.888467] Lustre: server umount lustre-OST0001 complete [ 5022.671450] Lustre: server umount lustre-MDT0000 complete [ 5024.453843] LustreError: 199413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570377 with bad export cookie 5924424312662158910 [ 5024.459134] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5024.469741] LustreError: 199413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5024.628798] Lustre: server umount lustre-MDT0001 complete [ 5035.687443] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:33:06 (1786570386) [ 5042.673382] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5046.574211] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5065.402882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.532505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5065.550523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5065.597385] Lustre: lustre-MDT0000: new disk, initializing [ 5065.646605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.657168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5067.863636] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5074.340732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.424561] Lustre: 207502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5074.446704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5074.449656] Lustre: Skipped 1 previous similar message [ 5074.502901] Lustre: lustre-MDT0001: new disk, initializing [ 5074.580397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5074.583621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5077.064810] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5082.633115] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5085.722292] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5089.816446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5089.984674] Lustre: lustre-OST0000: new disk, initializing [ 5089.988176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5089.992740] Lustre: 209468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5093.494136] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5098.016095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5098.023203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5098.104704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5099.184838] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5101.595149] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5101.696522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5103.842875] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5103.943235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5108.199419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5108.200682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.205339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5108.205346] Lustre: Skipped 7 previous similar messages [ 5108.223510] Lustre: Skipped 5 previous similar messages [ 5111.032757] Lustre: server umount lustre-OST0000 complete [ 5119.581914] Lustre: server umount lustre-MDT0000 complete [ 5121.574619] LustreError: 207495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570474 with bad export cookie 5924424312662160681 [ 5121.577871] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.582210] LustreError: 207495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5121.760489] Lustre: server umount lustre-MDT0001 complete [ 5125.991256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.208215] LustreError: 211681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5126.218191] LustreError: 211681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5128.343748] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5132.525321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.719433] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5138.417481] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5141.184505] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5144.414039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.385070] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5151.039897] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5153.233597] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5153.317905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5155.509219] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5158.694273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 5161.953769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5161.958929] LustreError: Skipped 1 previous similar message [ 5165.840873] Lustre: server umount lustre-OST0000 complete [ 5169.116101] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5171.513770] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5196.194891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.346607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.349393] Lustre: Skipped 5 previous similar messages [ 5199.089971] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5202.473288] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.479281] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5202.622410] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5203.475705] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5207.524405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.531861] Lustre: Skipped 4 previous similar messages [ 5207.537639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5207.539991] Lustre: Skipped 7 previous similar messages [ 5211.309404] Lustre: server umount lustre-OST0000 complete [ 5219.537322] Lustre: server umount lustre-MDT0000 complete [ 5221.353327] LustreError: 213520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570574 with bad export cookie 5924424312662161563 [ 5221.353841] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.360104] LustreError: 213520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5221.522697] Lustre: server umount lustre-MDT0001 complete [ 5225.528815] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:36:16 (1786570576) [ 5228.656468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.878900] LustreError: 217660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.886534] LustreError: 217660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5230.691376] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5234.400987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.510921] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5239.903540] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5242.206962] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5244.930106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5247.670920] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5251.215911] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5253.251742] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5254.369356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5256.634448] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5259.828734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 5263.400678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.484910] Lustre: lustre-OST0001: new disk, initializing [ 5263.487364] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5263.491250] Lustre: 221057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5266.325441] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5271.054515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5271.058997] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5271.120251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5271.276216] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5273.200712] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5273.290363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5274.979526] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5275.071148] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5276.133982] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5276.137352] LustreError: Skipped 2 previous similar messages [ 5278.020522] Lustre: server umount lustre-OST0001 complete [ 5281.013941] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5281.109230] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5283.117047] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5283.228946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5308.067163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.810634] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5313.514730] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5313.520703] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5313.522892] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5313.531787] Lustre: Skipped 1 previous similar message [ 5314.175939] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5314.864579] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5329.889343] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.940559] Lustre: server umount lustre-OST0001 complete [ 5344.225929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.230648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5344.233260] Lustre: Skipped 7 previous similar messages [ 5344.237596] Lustre: Skipped 15 previous similar messages [ 5348.567683] Lustre: server umount lustre-OST0000 complete [ 5356.456268] Lustre: server umount lustre-MDT0000 complete [ 5357.863218] LustreError: 217640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570710 with bad export cookie 5924424312662162508 [ 5357.865866] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.868040] LustreError: 217640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5372.384822] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5372.487339] Lustre: server umount lustre-MDT0001 complete [ 5381.857573] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:38:53 (1786570733) [ 5384.379934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.521728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5384.535419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5386.334222] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5390.013015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.079372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5390.091923] Lustre: 226288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5391.888626] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5394.986671] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5397.181704] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5399.756866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5399.853750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5402.099924] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5404.965798] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5409.064936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5409.093370] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5411.320171] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5414.265095] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5415.004809] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5416.777900] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5417.876178] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5419.557040] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5419.636507] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5421.905599] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d984703f000.ost_server_uuid 50 [ 5422.524028] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d984703f000.ost_server_uuid in FULL state after 0 sec [ 5422.901544] Lustre: Permanently deactivating lustre-OST0001 [ 5422.907161] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5424.027522] Lustre: Permanently reactivating lustre-OST0001 [ 5427.173614] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.176288] LustreError: Skipped 5 previous similar messages [ 5431.321706] Lustre: server umount lustre-OST0001 complete [ 5432.292035] LustreError: 228134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.299832] LustreError: 228134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 5439.098231] Lustre: server umount lustre-OST0000 complete [ 5455.328131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.416194] Lustre: server umount lustre-MDT0000 complete [ 5456.951368] LustreError: 228150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570809 with bad export cookie 5924424312662163796 [ 5456.955522] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5456.961682] LustreError: 228150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5457.112928] Lustre: server umount lustre-MDT0001 complete [ 5466.812303] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:40:18 (1786570818) [ 5469.967783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.081209] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5470.095751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5470.099323] Lustre: Skipped 5 previous similar messages [ 5470.175728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.178557] Lustre: Skipped 9 previous similar messages [ 5471.637509] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5474.677505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.716506] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5474.727174] Lustre: 232982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5476.548321] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5479.623134] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5481.926569] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5484.925813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5485.014398] Lustre: Permanently deactivating lustre-OST0000 [ 5485.020943] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5487.511513] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5490.253233] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5493.124130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5493.652929] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5493.707994] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5495.570814] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5498.365675] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.057838] Lustre: Permanently reactivating lustre-OST0000 [ 5502.014225] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5502.017401] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.021753] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5502.038603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.047341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5516.864118] Lustre: server umount lustre-OST0001 complete [ 5524.630568] Lustre: server umount lustre-OST0000 complete [ 5532.401631] Lustre: server umount lustre-MDT0000 complete [ 5534.280287] LustreError: 236564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570887 with bad export cookie 5924424312662165427 [ 5534.283391] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5534.285532] LustreError: 236564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5534.422777] Lustre: server umount lustre-MDT0001 complete [ 5539.171444] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5540.441645] Key type lgssc unregistered [ 5540.586383] LNet: 237850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5540.589646] LNetError: 237850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5540.596595] LNet: Removed LNI 192.168.202.106@tcp [ 5540.995857] Key type .llcrypt unregistered [ 5540.997412] Key type ._llcrypt unregistered [ 5551.288225] Key type ._llcrypt registered [ 5551.289721] Key type .llcrypt registered [ 5551.331823] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5551.674440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5551.711059] alg: No test for adler32 (adler32-zlib) [ 5552.599438] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 5552.696979] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 5554.280154] Key type lgssc registered [ 5554.716385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5559.008735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5559.020219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.236076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.796635] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5565.117675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5565.156813] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5565.167223] Lustre: 239286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5565.172579] Lustre: Permanently deactivating lustre-MDT0001 [ 5565.178489] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5565.296744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.999536] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5566.002578] LustreError: 239329:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5566.006663] LustreError: 239329:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5566.012665] LustreError: 239329:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 5566.834532] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5569.897237] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5572.252100] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5575.108064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.238427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.294246] LustreError: 241140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.304853] LustreError: 241140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5576.311696] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5577.398981] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5580.807857] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5581.793789] LustreError: 241140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.376455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.454870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5585.417821] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5585.514288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5585.517987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5585.550815] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5588.142049] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5589.889127] Lustre: Permanently reactivating lustre-MDT0001 [ 5589.896183] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5589.898814] Lustre: Skipped 2 previous similar messages [ 5595.812672] Lustre: lustre-MDT0001: Client c3952997-ded0-4b5d-a159-2d3644b64e66 (at 192.168.202.6@tcp) reconnecting [ 5598.247900] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5598.907706] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5600.857950] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5600.938284] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5602.610559] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5602.696742] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5603.874502] Lustre: Permanently deactivating lustre-MDT0001 [ 5603.882957] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5603.885181] Lustre: Skipped 2 previous similar messages [ 5612.038367] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5616.611160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.612081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5616.612865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5616.615984] Lustre: Skipped 3 previous similar messages [ 5621.732255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5621.735360] Lustre: Skipped 6 previous similar messages [ 5622.622792] Lustre: server umount lustre-MDT0000 complete [ 5624.082341] LustreError: 243775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570976 with bad export cookie 8580550272620749120 [ 5624.083180] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5624.087944] LustreError: 243775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5624.213661] Lustre: server umount lustre-MDT0001 complete [ 5631.812652] Lustre: server umount lustre-OST0000 complete [ 5636.040800] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:43:07 (1786570987) [ 5640.444329] Lustre: server umount lustre-OST0001 complete [ 5642.739091] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5646.342025] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5660.548221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.637502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5660.641795] Lustre: Skipped 2 previous similar messages [ 5660.650526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5660.695996] Lustre: lustre-MDT0000: new disk, initializing [ 5660.728143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.736429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5662.393305] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5667.038353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5667.085977] Lustre: 247640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5667.115118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5667.117528] Lustre: Skipped 1 previous similar message [ 5667.159156] Lustre: lustre-MDT0001: new disk, initializing [ 5667.209752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5667.214365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5668.788605] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5672.975502] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5675.216518] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5678.059889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.170049] Lustre: lustre-OST0000: new disk, initializing [ 5678.172163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5678.176333] Lustre: 249607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5678.221311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.227110] Lustre: Skipped 1 previous similar message [ 5680.546518] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5684.735794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5684.740458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5684.788405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5684.789995] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5685.560816] LustreError: 247648:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5690.039943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.113815] Lustre: lustre-OST0001: new disk, initializing [ 5690.116035] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5690.118912] Lustre: 250722:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5691.844039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5691.847594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5691.881980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5692.182784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5695.656110] LustreError: 247648:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5695.663694] LustreError: 247647:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5695.969609] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5696.497048] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5705.752803] LustreError: 247647:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5707.233943] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5707.236241] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.240087] LustreError: Skipped 1 previous similar message [ 5707.246848] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5707.253560] Lustre: Skipped 1 previous similar message [ 5712.355019] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5712.357664] Lustre: Skipped 1 previous similar message [ 5712.642731] Lustre: server umount lustre-OST0001 complete [ 5713.055070] LustreError: 251577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.475749] LustreError: 249592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.483077] LustreError: 249592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5722.599429] LustreError: 249590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.606545] LustreError: 249590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5725.665418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5725.665754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.667863] LustreError: Skipped 1 previous similar message [ 5725.671326] Lustre: Skipped 1 previous similar message [ 5725.672056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.678170] Lustre: Skipped 1 previous similar message [ 5732.836878] LustreError: 250305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.843146] LustreError: 250305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5737.955602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.957758] Lustre: Skipped 5 previous similar messages [ 5738.976122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5739.037420] Lustre: server umount lustre-OST0000 complete [ 5743.073726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.078243] Lustre: Skipped 2 previous similar messages [ 5746.786207] Lustre: server umount lustre-MDT0000 complete [ 5748.140329] LustreError: 247631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571101 with bad export cookie 8580550272620751899 [ 5748.143509] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5748.146334] LustreError: 247631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5754.298924] Lustre: server umount lustre-MDT0001 complete [ 5758.345870] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5759.483083] Key type lgssc unregistered [ 5759.617350] LNet: 252757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5759.620376] LNetError: 252757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5759.632947] LNet: Removed LNI 192.168.202.106@tcp [ 5759.941856] Key type .llcrypt unregistered [ 5759.943264] Key type ._llcrypt unregistered [ 5774.580997] Key type ._llcrypt registered [ 5774.582250] Key type .llcrypt registered [ 5774.627254] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5775.100613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.112216] alg: No test for adler32 (adler32-zlib) [ 5775.974772] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 5776.071117] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 5777.656177] Key type lgssc registered [ 5778.119341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5781.240196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5781.250706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.361257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5782.369496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5782.428336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.835310] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5786.980693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.018669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5787.031175] Lustre: 254939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5787.129197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.614652] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5791.242017] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5793.184860] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5795.686442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.771035] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5795.839106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.084719] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5800.934707] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5803.046130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5849.061055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5849.062181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.063405] LustreError: Skipped 1 previous similar message [ 5849.068316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5849.070420] Lustre: Skipped 1 previous similar message [ 5852.064857] Lustre: server umount lustre-OST0000 complete [ 5853.514789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5857.710088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.871953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5859.908910] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5862.484473] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5870.114102] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.118602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.118614] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5870.124519] Lustre: Skipped 1 previous similar message [ 5895.650407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.652144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.652225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.654957] Lustre: Skipped 2 previous similar messages [ 5895.658292] LustreError: Skipped 1 previous similar message [ 5900.770751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5900.773603] Lustre: Skipped 2 previous similar messages [ 5901.592921] Lustre: server umount lustre-OST0000 complete [ 5904.864786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5904.869096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.873514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.010711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5911.013896] Lustre: Skipped 3 previous similar messages [ 5917.152142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5917.218656] Lustre: server umount lustre-MDT0000 complete [ 5918.556498] LustreError: 254934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571271 with bad export cookie 12167210220664224411 [ 5918.559243] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.561921] LustreError: 254934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5918.684370] Lustre: server umount lustre-MDT0001 complete [ 5922.866266] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5923.974419] Key type lgssc unregistered [ 5924.115567] LNet: 260362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5924.118491] LNetError: 260362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5924.126452] LNet: Removed LNI 192.168.202.106@tcp [ 5924.480943] Key type .llcrypt unregistered [ 5924.482387] Key type ._llcrypt unregistered [ 5933.847302] Key type ._llcrypt registered [ 5933.848685] Key type .llcrypt registered [ 5933.890098] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5934.302747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5934.330330] alg: No test for adler32 (adler32-zlib) [ 5935.185995] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 5935.277833] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 5936.864137] Key type lgssc registered [ 5937.216490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5939.777230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5939.783713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.917323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5942.183734] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5944.904813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.023628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5946.288470] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5948.583496] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5950.357533] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5952.460685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5952.574551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.471210] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5956.891424] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5965.923948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5975.520813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5975.523360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.528277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.545820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.546197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.546392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.556160] Lustre: Skipped 1 previous similar message [ 5981.585357] Lustre: server umount lustre-OST0000 complete [ 5986.272885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5986.275817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.280555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.013537] Lustre: server umount lustre-MDT0000 complete [ 5990.323438] LustreError: 261736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571343 with bad export cookie 8336626643079526318 [ 5990.326412] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.327497] LustreError: 261736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5990.446546] Lustre: server umount lustre-MDT0001 complete [ 5994.111679] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5995.211563] Key type lgssc unregistered [ 5995.385624] LNet: 266078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5995.388327] LNetError: 266078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5995.400375] LNet: Removed LNI 192.168.202.106@tcp [ 5995.731727] Key type .llcrypt unregistered [ 5995.733203] Key type ._llcrypt unregistered [ 6002.973892] Key type ._llcrypt registered [ 6002.975302] Key type .llcrypt registered [ 6003.013239] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6003.417115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6003.430489] alg: No test for adler32 (adler32-zlib) [ 6004.290409] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6004.380945] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6005.960182] Key type lgssc registered [ 6006.332163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6009.001808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6009.008722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.152051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.605376] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6014.705629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.864254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6016.362419] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6018.750426] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6020.566911] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6022.969471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.082116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.959439] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6027.295794] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6035.367189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6045.669542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6045.669867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.672541] LustreError: Skipped 1 previous similar message [ 6045.679198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.682130] Lustre: Skipped 1 previous similar message [ 6047.443660] Lustre: server umount lustre-OST0000 complete [ 6050.787120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.788034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6050.791403] Lustre: Skipped 2 previous similar messages [ 6050.796940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6055.909266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6055.912403] Lustre: Skipped 2 previous similar messages [ 6061.025393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.028641] Lustre: Skipped 1 previous similar message [ 6063.072193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6063.164682] Lustre: server umount lustre-MDT0000 complete [ 6065.437695] LustreError: 267256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571418 with bad export cookie 6913882386208098913 [ 6065.441645] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6065.444401] LustreError: 267256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6065.675829] Lustre: server umount lustre-MDT0001 complete [ 6072.602685] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6074.683839] Key type lgssc unregistered [ 6074.846743] LNet: 270978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6074.851945] LNetError: 270978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6074.868616] LNet: Removed LNI 192.168.202.106@tcp [ 6075.461907] Key type .llcrypt unregistered [ 6075.464245] Key type ._llcrypt unregistered [ 6092.076653] Key type ._llcrypt registered [ 6092.079402] Key type .llcrypt registered [ 6092.207321] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6092.801411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6092.890262] alg: No test for adler32 (adler32-zlib) [ 6093.890813] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6094.063830] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6095.720144] Key type lgssc registered [ 6096.496905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6101.574489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6101.585768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.861165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.105983] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6111.634865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.926258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6115.139310] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6120.445486] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6124.306618] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6128.474538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6128.642816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6131.486113] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6135.390370] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6148.144434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6163.434619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.440169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6163.440954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.445861] Lustre: Skipped 1 previous similar message [ 6168.548693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.551256] Lustre: Skipped 1 previous similar message [ 6169.239423] Lustre: server umount lustre-OST0000 complete [ 6173.668673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.671334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6173.681118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6173.688415] Lustre: Skipped 2 previous similar messages [ 6177.895466] Lustre: server umount lustre-MDT0000 complete [ 6178.785831] LustreError: 272375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.793179] LustreError: 272375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6179.557567] LustreError: 272399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571532 with bad export cookie 4144012197925942208 [ 6179.557931] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.565787] LustreError: 272399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6179.732861] Lustre: server umount lustre-MDT0001 complete [ 6185.148130] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6186.620681] Key type lgssc unregistered [ 6186.773761] LNet: 276702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6186.777324] LNetError: 276702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6186.788695] LNet: Removed LNI 192.168.202.106@tcp [ 6187.155848] Key type .llcrypt unregistered [ 6187.159075] Key type ._llcrypt unregistered [ 6195.951547] Key type ._llcrypt registered [ 6195.953478] Key type .llcrypt registered [ 6196.017652] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6196.637702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6196.660525] alg: No test for adler32 (adler32-zlib) [ 6197.558551] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6197.675753] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6199.296306] Key type lgssc registered [ 6199.859657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6203.936215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6203.948159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.144723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6207.507578] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6212.652973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.888127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6215.253602] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6219.115437] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6222.145661] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6226.030783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.207745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.328907] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6230.326494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6233.061231] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.812352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.815378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6243.826639] Lustre: Skipped 1 previous similar message [ 6243.827039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.842359] Lustre: Skipped 1 previous similar message [ 6244.668782] Lustre: server umount lustre-OST0000 complete [ 6248.929292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6248.930306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.935548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6248.943917] Lustre: Skipped 1 previous similar message [ 6248.948747] Lustre: Skipped 1 previous similar message [ 6253.217565] Lustre: server umount lustre-MDT0000 complete [ 6254.049298] LustreError: 277905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.057546] LustreError: 277905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6255.423292] LustreError: 277882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571608 with bad export cookie 10445249321012997949 [ 6255.429992] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6255.434315] LustreError: 277882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6255.689342] Lustre: server umount lustre-MDT0001 complete [ 6261.707283] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6263.284138] Key type lgssc unregistered [ 6263.456277] LNet: 281615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6263.464195] LNetError: 281615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6264.489129] LNet: Removed LNI 192.168.202.106@tcp [ 6265.028817] Key type .llcrypt unregistered [ 6265.030495] Key type ._llcrypt unregistered [ 6271.469212] Key type ._llcrypt registered [ 6271.470779] Key type .llcrypt registered [ 6271.527717] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6281.115945] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6288.808286] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6289.161742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6289.169685] alg: No test for adler32 (adler32-zlib) [ 6290.028515] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6290.123327] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6291.704165] Key type lgssc registered [ 6292.115430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6305.232054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6305.239180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.335727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6306.352550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6306.392468] Lustre: lustre-MDT0000: new disk, initializing [ 6306.426390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6306.434540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6308.103263] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6313.921829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.989364] Lustre: 285138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6314.028879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6314.031428] Lustre: Skipped 1 previous similar message [ 6314.073403] Lustre: lustre-MDT0001: new disk, initializing [ 6314.130980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6314.154335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6314.160326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6316.630852] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6321.904476] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6324.353532] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6327.235287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6327.352137] Lustre: lustre-OST0000: new disk, initializing [ 6327.354835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6327.359338] Lustre: 287108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6327.399277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.816860] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6334.355872] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6335.498668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6335.502804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6335.557125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6336.201752] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6336.298862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6338.136162] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6338.223971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6340.578049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.579440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.584115] LustreError: Skipped 1 previous similar message [ 6340.584961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6340.589021] Lustre: Skipped 1 previous similar message [ 6345.113977] Lustre: server umount lustre-OST0000 complete [ 6350.304900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6350.307523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6350.312829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6350.315017] Lustre: Skipped 1 previous similar message [ 6352.681787] Lustre: server umount lustre-MDT0000 complete [ 6354.190301] LustreError: 285132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571707 with bad export cookie 16632134374767103307 [ 6354.193666] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6354.195687] LustreError: 285132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6354.326544] Lustre: server umount lustre-MDT0001 complete [ 6358.244834] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:55:09 (1786571709) [ 6360.891170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.094826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.862281] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6366.177795] LustreError: 289515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.183172] LustreError: 289515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6366.289157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.431919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6368.036416] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6371.131136] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6373.742878] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6376.567967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.675565] LustreError: 291332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.748868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6379.322781] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6382.770371] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6396.390822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.390983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6396.391532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.391538] Lustre: Skipped 1 previous similar message [ 6396.396411] Lustre: Skipped 2 previous similar messages [ 6398.688643] Lustre: server umount lustre-OST0000 complete [ 6401.505672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.512135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6401.515204] Lustre: Skipped 2 previous similar messages [ 6406.571854] Lustre: server umount lustre-MDT0000 complete [ 6406.625894] LustreError: 289516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.638208] LustreError: 289516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6408.323059] LustreError: 289497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571761 with bad export cookie 16632134374767104189 [ 6408.326046] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.327356] LustreError: 290250:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6408.327446] LustreError: 290250:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006ca889b2 x1873356146142464/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6408.328754] LustreError: 289497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6408.480478] Lustre: server umount lustre-MDT0001 complete [ 6412.881470] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6414.062560] Key type lgssc unregistered [ 6414.204755] LNet: 293047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6414.207377] LNetError: 293047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6414.214390] LNet: Removed LNI 192.168.202.106@tcp [ 6414.585322] Key type .llcrypt unregistered [ 6414.586392] Key type ._llcrypt unregistered [ 6425.758218] Key type ._llcrypt registered [ 6425.759943] Key type .llcrypt registered [ 6425.812792] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6433.364319] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6433.757568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6433.800675] alg: No test for adler32 (adler32-zlib) [ 6434.733603] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6434.843707] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6436.432153] Key type lgssc registered [ 6436.881092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6455.609627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6455.619438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6456.709614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6456.728758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6456.767867] Lustre: lustre-MDT0000: new disk, initializing [ 6456.803887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.816122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6458.386628] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6462.970496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.018364] Lustre: 296844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6463.043048] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6463.045191] Lustre: Skipped 1 previous similar message [ 6463.080479] Lustre: lustre-MDT0001: new disk, initializing [ 6463.117896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6463.133951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6463.137446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6464.664539] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6468.950845] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6470.965957] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6473.383532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6473.482648] Lustre: lustre-OST2710: new disk, initializing [ 6473.484426] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6473.487261] Lustre: 298813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6473.517446] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6475.805563] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6479.942127] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.921723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6481.925954] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6481.963418] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6482.348341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.409321] Lustre: lustre-OST03e8: new disk, initializing [ 6482.411624] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6482.414113] Lustre: 299876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6482.446055] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6484.425167] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6487.015841] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.069122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6488.073839] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6488.101565] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6489.725871] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6489.805877] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6491.206544] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6491.281274] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6492.761333] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6492.857649] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6494.239125] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6494.317386] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6498.275065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.277920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6498.281048] Lustre: Skipped 3 previous similar messages [ 6498.284014] Lustre: Skipped 2 previous similar messages [ 6501.480184] Lustre: server umount lustre-MDT0000 complete [ 6502.788420] LustreError: 297640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571855 with bad export cookie 16400685443268269699 [ 6502.791438] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6502.793334] LustreError: 297640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6502.910838] Lustre: server umount lustre-MDT0001 complete [ 6514.626946] Lustre: server umount lustre-OST2710 complete [ 6526.171589] Lustre: server umount lustre-OST03e8 complete [ 6532.554581] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6535.859902] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6547.793864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.876285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6547.885519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6547.921579] Lustre: lustre-MDT0000: new disk, initializing [ 6547.944297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.951074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6549.255818] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6553.596086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.639616] Lustre: 305116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6553.739845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6555.127429] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6558.884428] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6560.887678] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6563.330246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.434516] Lustre: lustre-OST0000: new disk, initializing [ 6563.436068] Lustre: Skipped 1 previous similar message [ 6563.437620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6563.441130] Lustre: Skipped 2 previous similar messages [ 6563.444164] Lustre: 307083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6563.476883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.480329] Lustre: Skipped 1 previous similar message [ 6565.545512] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6569.457988] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6570.975791] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6573.049678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6573.052953] Lustre: Skipped 1 previous similar message [ 6573.055173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6573.081266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6574.115225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6575.586841] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6575.659811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6578.146677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6578.149667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.154911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6578.157075] Lustre: Skipped 1 previous similar message [ 6582.489657] Lustre: server umount lustre-OST0000 complete [ 6584.801037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6584.803747] LustreError: Skipped 1 previous similar message [ 6584.805675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.810831] Lustre: Skipped 1 previous similar message [ 6584.812909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6588.385281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6588.385772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6593.505680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6593.508503] Lustre: Skipped 2 previous similar messages [ 6598.112343] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6598.180777] Lustre: server umount lustre-MDT0000 complete [ 6599.459094] LustreError: 305109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571952 with bad export cookie 16400685443268271365 [ 6599.462412] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6599.463276] LustreError: 305109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6599.590643] Lustre: server umount lustre-MDT0001 complete [ 6602.671253] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:59:13 (1786571953) [ 6603.189947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6603.798271] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:59:15 (1786571955) [ 6604.930781] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6614.455688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.545159] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6614.553832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6614.605775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.009706] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6619.037087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.078343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6619.087769] Lustre: 310955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6620.660519] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6637.027157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.091776] Lustre: Denying initial registration attempt from nid 192.168.202.106@tcp, specified as failover [ 6637.094776] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6637.097466] LustreError: 312407:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 6637.099941] LustreError: 312407:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6637.102677] LustreError: 312407:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6637.126482] Lustre: server umount lustre-OST0000 complete [ 6637.127994] LustreError: 312407:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6639.585075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6639.588094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.593596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6645.460274] Lustre: server umount lustre-MDT0000 complete [ 6646.710218] LustreError: 310945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571999 with bad export cookie 16400685443268272233 [ 6646.713216] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.714453] LustreError: 310945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.824662] Lustre: server umount lustre-MDT0001 complete [ 6650.330306] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6651.369920] Key type lgssc unregistered [ 6651.500328] LNet: 313528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6651.503669] LNetError: 313528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6651.515307] LNet: Removed LNI 192.168.202.106@tcp [ 6651.829263] Key type .llcrypt unregistered [ 6651.830562] Key type ._llcrypt unregistered [ 6655.362048] Key type ._llcrypt registered [ 6655.363471] Key type .llcrypt registered [ 6655.401906] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6655.830738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6655.837059] alg: No test for adler32 (adler32-zlib) [ 6656.698211] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6656.791735] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6658.384122] Key type lgssc registered [ 6658.757901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6683.774869] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6686.992825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6686.999379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6688.092376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6688.102129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6688.162386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.672616] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6692.972678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.014466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6693.031847] Lustre: 316269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6693.141102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.596825] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6697.475427] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6699.472932] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6714.851788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6714.922338] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6714.990806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.028068] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6719.490689] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6720.252690] Lustre: server umount lustre-OST0000 complete [ 6724.064930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6724.067916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6724.071962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6726.625205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6726.626462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.704180] Lustre: server umount lustre-MDT0000 complete [ 6729.130644] LustreError: 317008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572082 with bad export cookie 3539941795820741402 [ 6729.131636] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6729.134992] LustreError: 317008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6729.252765] Lustre: server umount lustre-MDT0001 complete [ 6733.139459] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6734.239813] Key type lgssc unregistered [ 6734.373338] LNet: 319851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6734.376077] LNetError: 319851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6734.382334] LNet: Removed LNI 192.168.202.106@tcp [ 6734.727156] Key type .llcrypt unregistered [ 6734.728347] Key type ._llcrypt unregistered [ 6744.688644] Key type ._llcrypt registered [ 6744.692805] Key type .llcrypt registered [ 6744.737604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6745.082795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6745.113314] alg: No test for adler32 (adler32-zlib) [ 6745.979728] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6746.080036] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6747.664156] Key type lgssc registered [ 6748.072246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6751.103963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6751.115230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.299034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.865855] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6756.962313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.090727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6758.531122] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6761.088799] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.085337] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.394937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6767.512414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.471098] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6771.903718] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6787.043031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6787.044189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6787.048223] Lustre: Skipped 2 previous similar messages [ 6787.050119] Lustre: Skipped 1 previous similar message [ 6789.796191] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 6789.799105] Lustre: Skipped 1 previous similar message [ 6792.162688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6792.167336] Lustre: Skipped 2 previous similar messages [ 6792.731878] Lustre: server umount lustre-MDT0000 complete [ 6794.095251] LustreError: 321225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572146 with bad export cookie 1521249391872991189 [ 6794.098270] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6794.100028] LustreError: 321225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6794.909169] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.915882] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6794.917951] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [ 6797.280990] LustreError: 321239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.281343] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.328854] LustreError: 321267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6799.329188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6799.335634] Lustre: Skipped 1 previous similar message [ 6800.238946] Lustre: server umount lustre-MDT0001 complete [ 6802.439767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6805.423931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.787205] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x151c9043115e7bab [ 6818.790377] Lustre: MGC192.168.202.106@tcp: Connection restored to 0@lo (at 0@lo) [ 6818.899340] LustreError: 323072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.907359] LustreError: 323072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6818.925484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.950989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6820.406725] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6823.400105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.536890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6824.549601] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6824.936735] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6827.384606] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6828.645333] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6830.943607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 6832.776449] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6833.327386] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6834.656740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6834.659358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6834.664374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6834.667434] Lustre: Skipped 1 previous similar message [ 6838.753107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6838.756107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.732526] Lustre: server umount lustre-OST0000 complete [ 6843.872831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6843.875735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6848.213910] Lustre: server umount lustre-MDT0000 complete [ 6849.506676] LustreError: 324860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.512507] LustreError: 324860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6849.549197] LustreError: 321224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572202 with bad export cookie 1521249391872998315 [ 6849.553366] LustreError: 321224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6849.553572] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.676679] Lustre: server umount lustre-MDT0001 complete [ 6853.445837] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6854.551492] Key type lgssc unregistered [ 6854.674329] LNet: 327526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6854.677654] LNetError: 327526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6854.685270] LNet: Removed LNI 192.168.202.106@tcp [ 6855.005192] Key type .llcrypt unregistered [ 6855.006422] Key type ._llcrypt unregistered [ 6864.332240] Key type ._llcrypt registered [ 6864.333347] Key type .llcrypt registered [ 6864.372183] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6864.771601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6864.835779] alg: No test for adler32 (adler32-zlib) [ 6865.685032] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6865.770375] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6867.352144] Key type lgssc registered [ 6867.709962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6870.165546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6870.171600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6871.308555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.616176] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6875.437923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.564562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.915794] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6879.430171] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6881.189904] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6885.283526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.287422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6885.289556] Lustre: Skipped 1 previous similar message [ 6888.363521] Lustre: server umount lustre-MDT0000 complete [ 6889.951396] LustreError: 328900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572242 with bad export cookie 1036455585076440576 [ 6889.951978] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.957461] LustreError: 328900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6890.119960] Lustre: server umount lustre-MDT0001 complete [ 6899.234049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.408849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6899.427153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6899.921567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.562211] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6907.890633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.952026] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6907.968874] Lustre: 331507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6908.124623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6910.967551] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6915.917721] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6919.390460] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.116840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.197928] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6926.204947] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6926.209814] LustreError: 333387:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6926.213181] LustreError: 333387:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6926.216273] LustreError: 333387:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6926.247681] Lustre: server umount lustre-OST0000 complete [ 6926.249594] LustreError: 333387:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6933.162528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.257935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6933.363670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.817840] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6939.200529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6941.464836] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6946.675454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.744745] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6946.811828] Lustre: lustre-OST0001: new disk, initializing [ 6946.814700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6946.819836] Lustre: 334929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6946.897341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.227614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6948.234135] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6948.331389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6951.269931] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6956.163098] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6958.564438] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6958.568365] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.582524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6958.589607] Lustre: Skipped 2 previous similar messages [ 6963.692910] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6963.699387] Lustre: Skipped 1 previous similar message [ 6968.803157] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6968.806668] Lustre: Skipped 1 previous similar message [ 6972.384171] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6972.438034] Lustre: server umount lustre-OST0001 complete [ 6973.933144] LustreError: 333853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.946385] LustreError: 333853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6974.945749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6974.946883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.953399] LustreError: Skipped 2 previous similar messages [ 6974.959759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6974.962959] Lustre: Skipped 2 previous similar messages [ 6974.975083] Lustre: Skipped 1 previous similar message [ 6979.044481] LustreError: 333854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.058845] LustreError: 333854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6980.724098] Lustre: server umount lustre-OST0000 complete [ 6984.161444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.162881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.171017] Lustre: Skipped 1 previous similar message [ 6984.176411] Lustre: Skipped 3 previous similar messages [ 6989.022552] Lustre: server umount lustre-MDT0000 complete [ 6989.282095] LustreError: 331514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.294290] LustreError: 331514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6990.953277] LustreError: 331497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572343 with bad export cookie 1036455585076441115 [ 6990.954051] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.962411] LustreError: 331497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6991.128581] Lustre: server umount lustre-MDT0001 complete [ 6997.431878] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6998.973519] Key type lgssc unregistered [ 6999.214491] LNet: 336890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6999.217455] LNetError: 336890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6999.237077] LNet: Removed LNI 192.168.202.106@tcp [ 6999.755343] Key type .llcrypt unregistered [ 6999.756823] Key type ._llcrypt unregistered [ 7039.625913] Key type ._llcrypt registered [ 7039.627668] Key type .llcrypt registered [ 7039.684052] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 7048.211583] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7048.831276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7048.870575] alg: No test for adler32 (adler32-zlib) [ 7049.759697] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7049.889216] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7051.512258] Key type lgssc registered [ 7052.086726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7068.599453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7068.611105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.753670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7069.770972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7069.822896] Lustre: lustre-MDT0000: new disk, initializing [ 7069.890895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.910386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7072.099484] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7077.245649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.293658] Lustre: 341795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7077.350653] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7077.352993] Lustre: Skipped 1 previous similar message [ 7077.389896] Lustre: lustre-MDT0001: new disk, initializing [ 7077.428956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.446076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7077.450376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7079.076468] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7083.440799] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7085.581113] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7088.353182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.497182] Lustre: lustre-OST0000: new disk, initializing [ 7088.499703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7088.507257] Lustre: 343764:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7088.563926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.314623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7090.319388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7090.382805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7091.515472] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7096.270375] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.141643] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7098.245477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7100.013920] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7100.096610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7100.900362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7100.900447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.903242] LustreError: Skipped 1 previous similar message [ 7100.907118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7100.909712] Lustre: Skipped 1 previous similar message [ 7106.019143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7106.021031] Lustre: Skipped 1 previous similar message [ 7111.140888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7111.143848] Lustre: Skipped 1 previous similar message [ 7115.232144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7115.314386] Lustre: server umount lustre-OST0000 complete [ 7118.304908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7118.308255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.313394] Lustre: Skipped 1 previous similar message [ 7118.315899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7121.379665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.113880] Lustre: server umount lustre-MDT0000 complete [ 7124.613307] LustreError: 343763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572477 with bad export cookie 8154215078899696530 [ 7124.613889] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7124.617823] LustreError: 343763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7124.755440] Lustre: server umount lustre-MDT0001 complete [ 7127.898051] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:07:59 (1786572479) [ 7130.582111] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:08:01 (1786572481) [ 7133.980718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.177173] LustreError: 346517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.184226] LustreError: 346517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7134.202555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.796739] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7139.099942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.238413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7140.814527] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7143.434972] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7145.438942] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7148.309355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.479660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7150.983537] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7154.082031] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7168.028447] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7168.651253] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7169.297340] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7169.929796] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7170.697363] Lustre: Failing over lustre-MDT0000 [ 7170.898055] Lustre: server umount lustre-MDT0000 complete [ 7174.822040] LustreError: 346514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7175.139948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.139985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7179.931944] LustreError: 346512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.939564] LustreError: 346512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7184.366454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.409908] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7184.507404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.531505] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7185.052741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7186.050358] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7189.989933] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7190.001901] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7190.020954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7192.305680] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7192.922909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7194.949842] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7195.694592] Lustre: Failing over lustre-MDT0000 [ 7195.894596] Lustre: server umount lustre-MDT0000 complete [ 7200.226623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7200.226685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.227742] LustreError: 346542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.227750] LustreError: 346542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7200.242940] Lustre: Skipped 3 previous similar messages [ 7205.345363] LustreError: 346517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7205.351387] LustreError: 346517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7209.278691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.335422] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7209.467246] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7210.651666] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7210.947321] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7214.569496] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7214.574711] Lustre: Skipped 2 previous similar messages [ 7214.590247] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7214.608151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7216.717517] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7217.367253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7224.801197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7224.803114] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7224.806425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7224.810025] Lustre: Skipped 2 previous similar messages [ 7224.812027] Lustre: Skipped 2 previous similar messages [ 7225.944379] Lustre: server umount lustre-MDT0000 complete [ 7227.363150] LustreError: 346497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572580 with bad export cookie 8154215078899699225 [ 7227.366437] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7227.367742] LustreError: 346497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7227.505610] Lustre: server umount lustre-MDT0001 complete [ 7238.277251] Lustre: server umount lustre-OST0000 complete [ 7244.990132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.183345] LustreError: 352867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.192766] LustreError: 352867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7245.219918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7245.224817] Lustre: Skipped 1 previous similar message [ 7247.052888] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7250.834705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.452609] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7255.555214] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7257.937343] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7261.277533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7264.518543] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7267.754253] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.658064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7282.146936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7282.147878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7282.151865] Lustre: Skipped 1 previous similar message [ 7282.156682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7282.159036] Lustre: Skipped 1 previous similar message [ 7285.790693] Lustre: server umount lustre-OST0000 complete [ 7293.744425] Lustre: server umount lustre-MDT0000 complete [ 7295.629293] LustreError: 352849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572648 with bad export cookie 8154215078899699974 [ 7295.634807] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.635185] LustreError: 352849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7295.797610] Lustre: server umount lustre-MDT0001 complete [ 7301.432772] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7302.585672] Key type lgssc unregistered [ 7302.738617] LNet: 356421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7302.741836] LNetError: 356421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7302.750366] LNet: Removed LNI 192.168.202.106@tcp [ 7303.144947] Key type .llcrypt unregistered [ 7303.146252] Key type ._llcrypt unregistered [ 7314.886283] Key type ._llcrypt registered [ 7314.887821] Key type .llcrypt registered [ 7314.941974] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7315.435128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7315.443187] alg: No test for adler32 (adler32-zlib) [ 7316.331366] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7316.453688] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7318.040191] Key type lgssc registered [ 7318.441943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7321.579593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7321.587498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.766529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.362699] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7328.050489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.215717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.775738] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7332.552654] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7334.722945] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7337.241912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.386365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.456982] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7342.021958] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7347.623118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7357.922025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.923584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7357.926027] Lustre: Skipped 2 previous similar messages [ 7357.929827] Lustre: Skipped 2 previous similar messages [ 7362.520166] Lustre: server umount lustre-MDT0000 complete [ 7363.043067] LustreError: 357911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.050486] LustreError: 357911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7363.818137] LustreError: 357896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572716 with bad export cookie 5020937938017877206 [ 7363.821517] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.823079] LustreError: 357896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7363.935839] Lustre: server umount lustre-MDT0001 complete [ 7375.449990] Lustre: server umount lustre-OST0000 complete [ 7380.006010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.164575] LustreError: 361641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.187888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.621664] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7384.739118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.892198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7386.363572] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7389.324104] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7391.541652] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7394.117947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7394.276331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7396.513107] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7399.206889] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7406.566733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7420.897572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7420.901528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.907292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7421.922321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7421.923277] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.073956] Lustre: server umount lustre-MDT0000 complete [ 7424.540944] LustreError: 361622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572777 with bad export cookie 5020937938017878305 [ 7424.542472] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7424.548309] LustreError: 361622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7424.662185] Lustre: server umount lustre-MDT0001 complete [ 7436.509335] Lustre: server umount lustre-OST0000 complete [ 7441.750211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.895104] LustreError: 365569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.919960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.300884] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7446.166949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.779866] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7450.521254] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7452.575794] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7454.884179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.142284] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7459.701339] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7466.279302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7471.586149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.586933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7471.587620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7471.587625] Lustre: Skipped 2 previous similar messages [ 7471.590580] Lustre: Skipped 2 previous similar messages [ 7476.706580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7476.708851] Lustre: Skipped 2 previous similar messages [ 7477.667817] Lustre: server umount lustre-OST0000 complete [ 7481.826301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.406019] Lustre: server umount lustre-MDT0000 complete [ 7486.777989] LustreError: 365549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572839 with bad export cookie 5020937938017879313 [ 7486.779709] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7486.784636] LustreError: 365549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7486.903679] Lustre: server umount lustre-MDT0001 complete [ 7490.853914] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7491.961811] Key type lgssc unregistered [ 7492.083362] LNet: 369123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7492.085882] LNetError: 369123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.095302] LNet: Removed LNI 192.168.202.106@tcp [ 7492.455993] Key type .llcrypt unregistered [ 7492.457371] Key type ._llcrypt unregistered [ 7502.687062] Key type ._llcrypt registered [ 7502.688098] Key type .llcrypt registered [ 7502.729300] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7503.079161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7503.122305] alg: No test for adler32 (adler32-zlib) [ 7503.985945] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7504.072379] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7505.656148] Key type lgssc registered [ 7506.044777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7508.970436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7508.976652] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.979785] LustreError: 370540:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7508.989122] LustreError: 370540:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7508.992234] LustreError: 370540:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7508.995572] LustreError: 370540:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7508.999180] LustreError: 370540:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7511.697872] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7511.700950] LustreError: 370936:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7511.709319] LustreError: 370936:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7511.712507] LustreError: 370936:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7511.715965] LustreError: 370936:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7511.719660] LustreError: 370936:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7518.288184] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7519.330104] Key type lgssc unregistered [ 7519.447443] LNet: 371939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7519.450331] LNetError: 371939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7519.456286] LNet: Removed LNI 192.168.202.106@tcp [ 7519.757346] Key type .llcrypt unregistered [ 7519.758562] Key type ._llcrypt unregistered [ 7526.111251] Key type ._llcrypt registered [ 7526.112421] Key type .llcrypt registered [ 7526.151037] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 7532.572986] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7532.906054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7532.914366] alg: No test for adler32 (adler32-zlib) [ 7533.785283] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7533.880679] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7535.464156] Key type lgssc registered [ 7535.875537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7548.746483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7548.753913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.832666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7549.843337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7549.878988] Lustre: lustre-MDT0000: new disk, initializing [ 7549.905760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7549.912386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7551.285506] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7555.680590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.729699] Lustre: 375069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7555.765695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7555.770352] Lustre: Skipped 1 previous similar message [ 7555.811747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7555.814740] Lustre: lustre-MDT0001: new disk, initializing [ 7555.853624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7555.867772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7555.870950] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7557.286533] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7561.129328] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7563.245112] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7565.919368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.037791] Lustre: lustre-OST0000: new disk, initializing [ 7566.039973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7566.043426] Lustre: 377037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7566.075556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7568.093146] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7571.941293] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.342582] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7573.493928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7573.497096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7573.532773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7574.441426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7575.837659] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7575.908612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7578.596912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.598071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7578.599145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7578.601343] Lustre: Skipped 1 previous similar message [ 7582.683062] Lustre: server umount lustre-OST0000 complete [ 7586.785049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7586.788141] LustreError: Skipped 1 previous similar message [ 7586.789555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.794565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7586.796894] Lustre: Skipped 1 previous similar message [ 7588.832929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7588.833609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.183378] Lustre: server umount lustre-MDT0000 complete [ 7591.464172] LustreError: 375879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572944 with bad export cookie 1878895745439376538 [ 7591.467516] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7591.467607] LustreError: 375879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7591.595267] Lustre: server umount lustre-MDT0001 complete [ 7594.445617] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7595.049331] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:15:46 (1786572946) [ 7597.544432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7597.741319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7599.206437] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7602.322225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7602.466512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7603.929378] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7606.548801] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7608.444156] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7610.854221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7610.962656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.075461] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7615.765267] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7618.203551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7618.274976] Lustre: lustre-OST0001: new disk, initializing [ 7618.277112] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7618.280136] Lustre: 382330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7620.338233] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7623.177548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7623.181469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7623.217528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7624.311300] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7628.260876] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7628.263229] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.266124] LustreError: Skipped 1 previous similar message [ 7628.272181] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7628.274955] Lustre: Skipped 2 previous similar messages [ 7632.324844] Lustre: server umount lustre-OST0001 complete [ 7633.379337] LustreError: 382929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.385224] LustreError: 382929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7638.499218] LustreError: 381318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.505147] LustreError: 381318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7643.620937] LustreError: 381315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.628628] LustreError: 381315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7648.224800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7648.229051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7648.234153] Lustre: Skipped 1 previous similar message [ 7648.236681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7648.239389] Lustre: Skipped 1 previous similar message [ 7648.738731] LustreError: 383263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7648.745397] LustreError: 383263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7650.643724] Lustre: server umount lustre-OST0000 complete [ 7653.857915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7658.277183] Lustre: server umount lustre-MDT0000 complete [ 7658.976990] LustreError: 379498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.985939] LustreError: 379498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7659.545287] LustreError: 382329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573012 with bad export cookie 1878895745439377420 [ 7659.549059] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.550114] LustreError: 382329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7665.723906] Lustre: server umount lustre-MDT0001 complete [ 7669.547123] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7670.588215] Key type lgssc unregistered [ 7670.722353] LNet: 384317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7670.725263] LNetError: 384317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7670.732280] LNet: Removed LNI 192.168.202.106@tcp [ 7671.031378] Key type .llcrypt unregistered [ 7671.032921] Key type ._llcrypt unregistered [ 7689.102250] Key type ._llcrypt registered [ 7689.103528] Key type .llcrypt registered [ 7689.145192] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7689.522625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7689.561607] alg: No test for adler32 (adler32-zlib) [ 7690.410394] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7690.498673] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7692.080160] Key type lgssc registered [ 7692.446326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7695.210874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7695.222233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.317498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7696.325538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7696.383351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.797426] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7700.864794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.909560] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7700.922133] Lustre: 387091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7701.030216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7702.727541] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7705.770751] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7708.198805] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7709.665096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.668156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7709.671951] Lustre: Skipped 1 previous similar message [ 7714.785274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7714.787877] Lustre: Skipped 2 previous similar messages [ 7715.353044] Lustre: server umount lustre-MDT0000 complete [ 7716.820260] LustreError: 387128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573069 with bad export cookie 8923194816647639121 [ 7716.823512] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7716.824626] LustreError: 387128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7716.935983] Lustre: server umount lustre-MDT0001 complete [ 7720.692684] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:17:52 (1786573072) [ 7723.333620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7723.542865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7725.091194] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7728.161943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.284322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7729.645772] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7732.051261] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7734.009377] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7736.305699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7736.386201] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7736.471078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.677739] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7741.269854] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7744.967035] Lustre: Permanently deactivating lustre-OST0000 [ 7744.972776] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7745.281269] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7746.838977] LustreError: 392443:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7746.843080] LustreError: 392443:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7746.845714] LustreError: 392443:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7747.423548] LustreError: 392491:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7747.426265] LustreError: 392491:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7762.400138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7762.447941] Lustre: server umount lustre-OST0000 complete [ 7763.331031] LustreError: 392741:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7763.333545] LustreError: 392741:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7764.450383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7764.453669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.459641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.521054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.521505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7772.641038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.643468] Lustre: Skipped 2 previous similar messages [ 7778.272185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7778.326661] Lustre: server umount lustre-MDT0000 complete [ 7779.549034] LustreError: 389581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573132 with bad export cookie 8923194816647639688 [ 7779.552399] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7779.554159] LustreError: 389581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7790.092195] Lustre: server umount lustre-MDT0001 complete [ 7792.816890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.141231] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7795.015857] LustreError: 393899:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7797.159385] Lustre: 394237:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.106@tcp [ 7797.887892] Lustre: server umount MGS complete [ 7801.604625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.758206] LustreError: 394961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.764469] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7801.765963] LustreError: 394961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7801.768126] Lustre: Skipped 1 previous similar message [ 7801.786844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7803.315757] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7806.172413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.291169] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7807.800933] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7810.171525] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7811.903736] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7812.573298] Lustre: Permanently reactivating lustre-OST0000 [ 7812.578918] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7812.582420] Lustre: Skipped 2 previous similar messages [ 7836.128097] LustreError: 396691:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7838.628428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7838.757485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.761519] Lustre: Skipped 1 previous similar message [ 7840.726239] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7843.460578] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7856.096993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7856.099326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.103386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7856.105883] Lustre: Skipped 2 previous similar messages [ 7857.123298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7861.708218] Lustre: server umount lustre-OST0000 complete [ 7867.361511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7867.366026] Lustre: Skipped 2 previous similar messages [ 7869.012447] Lustre: server umount lustre-MDT0000 complete [ 7870.176335] LustreError: 396533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573223 with bad export cookie 8923194816647640710 [ 7870.177042] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7870.180149] LustreError: 396533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7870.298151] Lustre: server umount lustre-MDT0001 complete [ 7873.767477] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7874.785555] Key type lgssc unregistered [ 7874.912542] LNet: 399537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7874.915262] LNetError: 399537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7874.923344] LNet: Removed LNI 192.168.202.106@tcp [ 7875.233189] Key type .llcrypt unregistered [ 7875.234398] Key type ._llcrypt unregistered [ 7881.436813] Key type ._llcrypt registered [ 7881.437870] Key type .llcrypt registered [ 7881.474162] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 7888.012546] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7888.452192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7888.459721] alg: No test for adler32 (adler32-zlib) [ 7889.314220] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7889.406359] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7890.984262] Key type lgssc registered [ 7891.349378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7903.059024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7903.065714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.158473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7904.170717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7904.210484] Lustre: lustre-MDT0000: new disk, initializing [ 7904.248118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.255429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7905.643526] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7909.751807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.796042] Lustre: 402665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7909.818490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7909.820835] Lustre: Skipped 1 previous similar message [ 7909.851619] Lustre: lustre-MDT0001: new disk, initializing [ 7909.880909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7909.893452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7909.897139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7911.353656] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7914.984269] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7916.840143] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7919.044732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7919.144617] Lustre: lustre-OST0000: new disk, initializing [ 7919.146908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7919.149799] Lustre: 404634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7919.181163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7920.999552] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7921.339040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7921.343268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7921.387134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7924.614349] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7925.970863] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7926.042892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7927.356939] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7927.426620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7931.874409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.874885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7931.876291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7931.878979] Lustre: Skipped 1 previous similar message [ 7931.881829] LustreError: Skipped 1 previous similar message [ 7934.163664] Lustre: server umount lustre-OST0000 complete [ 7935.456829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7935.459523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.464571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.466456] Lustre: Skipped 1 previous similar message [ 7936.993327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7936.993866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.599716] Lustre: server umount lustre-MDT0000 complete [ 7945.072374] LustreError: 403461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573297 with bad export cookie 10703869859345920586 [ 7945.075190] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7945.076681] LustreError: 403461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7945.209437] Lustre: server umount lustre-MDT0001 complete [ 7948.453939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7949.072265] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:21:40 (1786573300) [ 7951.370671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7951.519171] LustreError: 407089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7951.531746] LustreError: 407089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7951.562977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.986948] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7955.974506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.106200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7957.504184] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7960.237645] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7962.203605] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7964.666653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.788983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.825442] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7969.600730] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7970.317617] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7981.289569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7986.657495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7986.660220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7986.664909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7986.667105] Lustre: Skipped 1 previous similar message [ 7988.883510] Lustre: server umount lustre-OST0000 complete [ 7992.289080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7992.291992] LustreError: Skipped 1 previous similar message [ 7992.293851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.298944] Lustre: Skipped 1 previous similar message [ 7992.301220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7992.303582] Lustre: Skipped 2 previous similar messages [ 7996.571034] Lustre: server umount lustre-MDT0000 complete [ 7997.846319] LustreError: 408929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573350 with bad export cookie 10703869859345921454 [ 7997.849644] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7997.851583] LustreError: 408929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7997.971503] Lustre: server umount lustre-MDT0001 complete [ 8001.536240] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8002.556463] Key type lgssc unregistered [ 8002.678465] LNet: 410836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8002.680841] LNetError: 410836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8002.693332] LNet: Removed LNI 192.168.202.106@tcp [ 8003.015274] Key type .llcrypt unregistered [ 8003.016248] Key type ._llcrypt unregistered [ 8010.416697] Key type ._llcrypt registered [ 8010.417921] Key type .llcrypt registered [ 8010.452990] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8010.567340] Key type .llcrypt unregistered [ 8010.568453] Key type ._llcrypt unregistered [ 8017.249582] Key type ._llcrypt registered [ 8017.251332] Key type .llcrypt registered [ 8017.286197] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8017.601412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.630594] alg: No test for adler32 (adler32-zlib) [ 8018.483798] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8018.567237] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8020.152142] Key type lgssc registered [ 8020.523552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8023.198258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8023.203831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8024.353829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.654709] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8028.498562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8028.608362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.420358] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8032.693828] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.761718] LustreError: 413026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.768027] LustreError: 413026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8034.785549] LustreError: 413045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8034.790237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8034.792923] LustreError: 413045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8034.821653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.934626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8036.227132] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8045.024842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8045.028266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.032735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8045.537697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8045.540990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.546802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8049.302661] Lustre: server umount lustre-OST0000 complete [ 8050.657464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8050.657565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8050.658147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.667203] Lustre: Skipped 1 previous similar message [ 8055.777100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.779465] Lustre: Skipped 2 previous similar messages [ 8056.661382] Lustre: server umount lustre-MDT0000 complete [ 8057.944513] LustreError: 413052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573410 with bad export cookie 14077720059966841181 [ 8057.947422] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8057.948779] LustreError: 413052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8058.086497] Lustre: server umount lustre-MDT0001 complete [ 8061.656856] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8062.643612] Key type lgssc unregistered [ 8062.763353] LNet: 416176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8062.765579] LNetError: 416176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8062.772251] LNet: Removed LNI 192.168.202.106@tcp [ 8063.078619] Key type .llcrypt unregistered [ 8063.080078] Key type ._llcrypt unregistered [ 8071.956875] Key type ._llcrypt registered [ 8071.958822] Key type .llcrypt registered [ 8072.002707] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8072.409496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8072.429199] alg: No test for adler32 (adler32-zlib) [ 8073.269828] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8073.352088] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8074.928152] Key type lgssc registered [ 8075.261163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8077.701295] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8077.707117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8093.152330] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8098.272207] LustreError: 417543:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [ 8124.896224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.654735] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8128.968680] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8131.238131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.399425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8131.423926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8132.681477] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8135.375531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.496964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8136.795588] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8139.126206] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8140.841832] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8144.866300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.868374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.870772] Lustre: Skipped 1 previous similar message [ 8144.874733] Lustre: Skipped 1 previous similar message [ 8149.265330] Lustre: server umount lustre-OST0000 complete [ 8151.008889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8151.012457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8151.018434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.105144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.105678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8156.625577] Lustre: server umount lustre-MDT0000 complete [ 8157.830979] LustreError: 417549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573510 with bad export cookie 7923239773692761599 [ 8157.834469] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.835037] LustreError: 417549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8157.944147] Lustre: server umount lustre-MDT0001 complete [ 8161.575035] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8162.564519] Key type lgssc unregistered [ 8162.690254] LNet: 421111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8162.693483] LNetError: 421111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8162.702274] LNet: Removed LNI 192.168.202.106@tcp [ 8162.998652] Key type .llcrypt unregistered [ 8162.999901] Key type ._llcrypt unregistered [ 8172.290127] Key type ._llcrypt registered [ 8172.291234] Key type .llcrypt registered [ 8172.323741] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8172.635599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8172.663096] alg: No test for adler32 (adler32-zlib) [ 8173.531285] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8173.625080] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8175.208169] Key type lgssc registered [ 8175.570837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8178.146089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8178.153540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.296100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.591822] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8183.422548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.553033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8185.039861] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8187.474360] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8189.240869] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8191.407903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8191.506524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8193.331947] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8195.729627] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8204.839690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8209.377167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8209.379595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.383476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.891031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.891652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.894774] Lustre: Skipped 1 previous similar message [ 8212.644105] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 8212.646498] Lustre: Skipped 1 previous similar message [ 8214.561742] Lustre: server umount lustre-MDT0000 complete [ 8215.009176] LustreError: 423232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.013789] LustreError: 423232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8215.376575] LustreError: 424345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.6@tcp arrived at 1786573568 with bad export cookie 15076429989451911629 [ 8215.382574] LustreError: 424345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8216.034154] LustreError: 422484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573568 with bad export cookie 15076429989451910978 [ 8216.038017] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.038639] LustreError: 422484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8219.104931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8219.108730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.115138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8220.129066] LustreError: 422504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.134973] LustreError: 422504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8221.153492] LustreError: 423232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.158420] LustreError: 423232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8222.171867] Lustre: server umount lustre-OST0000 complete [ 8224.514524] Lustre: server umount lustre-MDT0001 complete [ 8228.014710] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8229.005472] Key type lgssc unregistered [ 8229.128344] LNet: 426208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8229.130446] LNetError: 426208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8229.143300] LNet: Removed LNI 192.168.202.106@tcp [ 8229.438337] Key type .llcrypt unregistered [ 8229.439819] Key type ._llcrypt unregistered [ 8238.291587] Key type ._llcrypt registered [ 8238.293101] Key type .llcrypt registered [ 8238.329080] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8238.612781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.624937] alg: No test for adler32 (adler32-zlib) [ 8239.483463] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8239.592500] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8241.176167] Key type lgssc registered [ 8241.548849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8244.059420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8244.066853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.204703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.495285] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8249.170729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.288622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8250.560479] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8252.843863] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8254.526939] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.670962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.776792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.634103] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8260.907520] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8271.143438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8274.912858] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8274.915326] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8274.919703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8276.449350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8276.451816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.605463] Lustre: server umount lustre-MDT0001 complete [ 8281.569469] LustreError: 428326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8281.569943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8281.570130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.570503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8281.570506] Lustre: Skipped 2 previous similar messages [ 8281.573774] LustreError: 428326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8286.149435] Lustre: server umount lustre-OST0000 complete [ 8286.689715] LustreError: 427600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.347240] Lustre: server umount lustre-MDT0000 complete [ 8291.257481] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8292.160213] Key type lgssc unregistered [ 8292.287387] LNet: 431305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8292.290562] LNetError: 431305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8292.299362] LNet: Removed LNI 192.168.202.106@tcp [ 8292.575829] Key type .llcrypt unregistered [ 8292.576844] Key type ._llcrypt unregistered [ 8300.212952] Key type ._llcrypt registered [ 8300.214409] Key type .llcrypt registered [ 8300.252511] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 8306.343960] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8306.644284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8306.665296] alg: No test for adler32 (adler32-zlib) [ 8307.522233] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8307.612751] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8309.192122] Key type lgssc registered [ 8309.607781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8321.810855] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8326.946085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8326.952018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.025520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8328.036093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8328.069941] Lustre: lustre-MDT0000: new disk, initializing [ 8328.090623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.095859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8329.380976] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8335.073304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8335.170753] Lustre: lustre-OST0000: new disk, initializing [ 8335.172317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8335.174276] Lustre: Skipped 1 previous similar message [ 8335.176222] Lustre: 436314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8335.197591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8337.071160] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8339.966501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8339.971046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8339.979247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8340.774121] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8346.009911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.052401] Lustre: 435292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8346.079826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8346.114828] Lustre: lustre-MDT0001: new disk, initializing [ 8346.151554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8346.164440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8346.167460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8347.427602] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8351.010602] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8351.077866] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8352.377869] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8352.450629] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8355.988401] Lustre: server umount lustre-MDT0001 complete [ 8358.881771] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8358.882866] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.890341] Lustre: Skipped 1 previous similar message [ 8364.002173] LustreError: 435301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8364.007747] LustreError: 435301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8367.010803] LustreError: 435329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8367.016253] LustreError: 435329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8370.656864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.659049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.663569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8372.194151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8372.194846] LustreError: 435305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.296478] Lustre: server umount lustre-OST0000 complete [ 8381.748537] Lustre: server umount lustre-MDT0000 complete [ 8385.937962] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8386.925566] Key type lgssc unregistered [ 8387.052267] LNet: 440056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8387.054436] LNetError: 440056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8387.062260] LNet: Removed LNI 192.168.202.106@tcp [ 8387.364315] Key type .llcrypt unregistered [ 8387.365481] Key type ._llcrypt unregistered [ 8399.511639] Key type ._llcrypt registered [ 8399.512997] Key type .llcrypt registered [ 8399.548138] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8399.647428] Key type .llcrypt unregistered [ 8399.648903] Key type ._llcrypt unregistered [ 8406.009251] Key type ._llcrypt registered [ 8406.010387] Key type .llcrypt registered [ 8406.044142] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8406.401496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8406.430808] alg: No test for adler32 (adler32-zlib) [ 8407.276135] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8407.362769] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8408.944164] Key type lgssc registered [ 8409.293305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8417.760452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8417.765793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.843462] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8418.852906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8418.888741] Lustre: lustre-MDT0000: new disk, initializing [ 8418.906926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8418.913603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8420.186406] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8424.297454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.345233] Lustre: 443410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8424.375523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8424.377210] Lustre: Skipped 1 previous similar message [ 8424.410091] Lustre: lustre-MDT0001: new disk, initializing [ 8424.441489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8424.456546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8424.460690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8426.087321] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8431.660521] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8435.042875] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8438.938283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.067355] Lustre: lustre-OST0000: new disk, initializing [ 8439.069578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8439.073355] Lustre: 445375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8439.124973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8442.507784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8448.050604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8448.061429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8448.145146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8448.879332] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8455.137389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8455.145367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.166198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.723102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.726790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.735151] Lustre: Skipped 1 previous similar message [ 8458.737867] Lustre: Skipped 1 previous similar message [ 8460.315751] Lustre: server umount lustre-MDT0000 complete [ 8463.380473] LustreError: 443402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573816 with bad export cookie 10917541699536141229 [ 8463.384761] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8463.387014] LustreError: 443402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8463.709159] Lustre: server umount lustre-MDT0001 complete [ 8477.099598] Lustre: server umount lustre-OST0000 complete [ 8484.264113] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:30:35 (1786573835) [ 8489.576834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.934554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8492.515428] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8495.075164] LustreError: 447401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.079631] LustreError: 447401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8497.190603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.379232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8499.778243] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8503.506749] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8506.059342] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8521.978632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.124316] Lustre: Found index 0 for lustre-OST0000, updating log [ 8522.224899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.484827] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8529.541922] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8535.531197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8543.713709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8543.716569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.724205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8543.728595] Lustre: Skipped 1 previous similar message [ 8548.200942] Lustre: server umount lustre-OST0000 complete [ 8564.193374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8564.197473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.202510] Lustre: Skipped 1 previous similar message [ 8564.205141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8564.207616] Lustre: Skipped 3 previous similar messages [ 8567.687734] Lustre: server umount lustre-MDT0000 complete [ 8569.594887] LustreError: 447381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573922 with bad export cookie 10917541699536142608 [ 8569.597014] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8569.603187] LustreError: 447381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8569.764565] Lustre: server umount lustre-MDT0001 complete [ 8574.226636] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:32:05 (1786573925) [ 8580.448914] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [ 9168.402681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.647746] LustreError: 453826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9168.717899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.310286] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9173.985189] LustreError: 453827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.552278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.821155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9179.755584] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9184.479856] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9187.900863] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9204.971540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9205.133125] Lustre: Found index 0 for lustre-OST0000, updating log [ 9205.262354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9208.671858] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9213.270363] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9214.594477] Lustre: server umount lustre-OST0000 complete [ 9218.018020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9218.020420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.025522] Lustre: Skipped 1 previous similar message [ 9218.028234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9218.031694] Lustre: Skipped 2 previous similar messages [ 9220.448441] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [ 9220.579165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.579442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.652741] Lustre: server umount lustre-MDT0000 complete [ 9224.739945] LustreError: 454554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9224.747841] LustreError: 454554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9224.969493] LustreError: 453806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786574577 with bad export cookie 10917541699536143693 [ 9224.976113] LustreError: 453806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9224.977199] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9225.148673] Lustre: server umount lustre-MDT0001 complete [ 9247.610775] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9252.627857] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9271.112635] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:43:42 (1786574622) [ 9280.780661] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9282.450289] Key type lgssc unregistered [ 9284.705282] LNet: 460430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9284.712810] LNetError: 460430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9335.910299] LNet: Removed LNI 192.168.202.106@tcp [ 9336.501847] Key type .llcrypt unregistered [ 9336.503761] Key type ._llcrypt unregistered [ 9347.718265] Key type ._llcrypt registered [ 9347.719336] Key type .llcrypt registered [ 9347.781815] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9348.474471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9348.601436] alg: No test for adler32 (adler32-zlib) [ 9349.661790] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9349.822220] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9351.448744] Key type lgssc registered [ 9352.077710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9354.535502] LNet: Added LNI 192.168.202.106@tcp100 [8/256/0/180] [ 9355.089590] LNet: Added LNI 192.168.202.106@tcp101 [8/256/0/180] [ 9355.698827] LNet: Added LNI 192.168.202.106@tcp102 [8/256/0/180] [ 9356.216391] LNet: Added LNI 192.168.202.106@tcp103 [8/256/0/180] [ 9356.690201] LNet: Added LNI 192.168.202.106@tcp104 [8/256/0/180] [ 9357.179895] LNet: Added LNI 192.168.202.106@tcp105 [8/256/0/180] [ 9357.669692] LNet: Added LNI 192.168.202.106@tcp106 [8/256/0/180] [ 9358.162354] LNet: Added LNI 192.168.202.106@tcp107 [8/256/0/180] [ 9358.741592] LNet: Added LNI 192.168.202.106@tcp108 [8/256/0/180] [ 9359.203589] LNet: Added LNI 192.168.202.106@tcp109 [8/256/0/180] [ 9359.716204] LNet: Added LNI 192.168.202.106@tcp110 [8/256/0/180] [ 9360.173372] LNet: Added LNI 192.168.202.106@tcp111 [8/256/0/180] [ 9360.701233] LNet: Added LNI 192.168.202.106@tcp112 [8/256/0/180] [ 9361.194115] LNet: Added LNI 192.168.202.106@tcp113 [8/256/0/180] [ 9361.658297] LNet: Added LNI 192.168.202.106@tcp114 [8/256/0/180] [ 9362.149709] LNet: Added LNI 192.168.202.106@tcp115 [8/256/0/180] [ 9362.673948] LNet: Added LNI 192.168.202.106@tcp116 [8/256/0/180] [ 9363.170966] LNet: Added LNI 192.168.202.106@tcp117 [8/256/0/180] [ 9363.614563] LNet: Added LNI 192.168.202.106@tcp118 [8/256/0/180] [ 9364.129200] LNet: Added LNI 192.168.202.106@tcp119 [8/256/0/180] [ 9364.663105] LNet: Added LNI 192.168.202.106@tcp120 [8/256/0/180] [ 9365.208407] LNet: Added LNI 192.168.202.106@tcp121 [8/256/0/180] [ 9365.710717] LNet: Added LNI 192.168.202.106@tcp122 [8/256/0/180] [ 9366.213558] LNet: Added LNI 192.168.202.106@tcp123 [8/256/0/180] [ 9366.715343] LNet: Added LNI 192.168.202.106@tcp124 [8/256/0/180] [ 9367.190169] LNet: Added LNI 192.168.202.106@tcp125 [8/256/0/180] [ 9367.648661] LNet: Added LNI 192.168.202.106@tcp126 [8/256/0/180] [ 9368.148686] LNet: Added LNI 192.168.202.106@tcp127 [8/256/0/180] [ 9368.729693] LNet: Added LNI 192.168.202.106@tcp128 [8/256/0/180] [ 9369.273994] LNet: Added LNI 192.168.202.106@tcp129 [8/256/0/180] [ 9369.756235] LNet: Added LNI 192.168.202.106@tcp130 [8/256/0/180] [ 9370.310582] LNet: Added LNI 192.168.202.106@tcp131 [8/256/0/180] [ 9370.831814] LNet: Added LNI 192.168.202.106@tcp132 [8/256/0/180] [ 9371.308744] LNet: Added LNI 192.168.202.106@tcp133 [8/256/0/180] [ 9371.881205] LNet: Added LNI 192.168.202.106@tcp134 [8/256/0/180] [ 9372.421982] LNet: Added LNI 192.168.202.106@tcp135 [8/256/0/180] [ 9376.277640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9376.301301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.363528] Lustre: 463426:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [ 9377.417610] LNetError: 463451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9377.427804] LNetError: Protocol error connecting to 192.168.202.106@tcp119 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9377.438067] LNetError: Skipped 7 previous similar messages [ 9377.500801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9377.505423] LNet: 5 local NIs in recovery (showing 5): 192.168.202.106@tcp125, 192.168.202.106@tcp126, 192.168.202.106@tcp128, 192.168.202.106@tcp129, 192.168.202.106@tcp119 [ 9377.519850] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9377.573575] Lustre: lustre-MDT0000: new disk, initializing [ 9377.622864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.636801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9378.466386] LNetError: 463458:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9378.479305] LNetError: 463458:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 9378.486221] LNetError: Protocol error connecting to 192.168.202.106@tcp118 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9378.495351] LNetError: Skipped 1 previous similar message [ 9379.674723] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9380.578418] LNetError: 463447:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9380.592055] LNetError: 463447:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 9380.600323] LNetError: Protocol error connecting to 192.168.202.106@tcp126 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9380.611081] LNetError: Skipped 5 previous similar messages [ 9384.674104] LNetError: 463461:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9384.686527] LNetError: Protocol error connecting to 192.168.202.106@tcp126 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9386.488794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.629589] Lustre: 463442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9386.697331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9386.700135] Lustre: Skipped 1 previous similar message [ 9386.763468] Lustre: lustre-MDT0001: new disk, initializing [ 9386.804949] Lustre: 464248:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [ 9386.812261] Lustre: 464248:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9386.884579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9386.931918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9386.942642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9389.826720] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9392.874866] LNetError: 463458:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9392.882214] LNetError: Protocol error connecting to 192.168.202.106@tcp126 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9394.698941] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [ 9394.706028] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9396.361206] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9399.667533] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9403.894381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9404.078500] Lustre: lustre-OST0000: new disk, initializing [ 9404.080991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9404.085447] Lustre: 465418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9404.143701] Lustre: 465423:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [ 9404.147746] Lustre: 465423:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 9404.185745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.743844] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9408.228773] LNetError: 463451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9408.234266] LNetError: Protocol error connecting to 192.168.202.106@tcp126 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9410.555612] Lustre: 465936:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [ 9410.559409] Lustre: 465936:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9410.588488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9410.594168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9410.657788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9413.364306] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9420.772472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.774857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9420.801257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.544187] Lustre: server umount lustre-OST0000 complete [ 9425.890272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9425.891392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9425.901852] Lustre: Skipped 2 previous similar messages [ 9425.912474] Lustre: Skipped 2 previous similar messages [ 9427.681441] LNetError: 460889:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [ 9427.686665] LNetError: 460889:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 9427.690210] LNetError: Protocol error connecting to 192.168.202.106@tcp123 at host 192.168.202.106:988: is it running a compatible version of Lustre? [ 9427.695273] LNetError: Skipped 9 previous similar messages [ 9431.011081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9431.017282] Lustre: Skipped 1 previous similar message [ 9436.130604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9436.132845] Lustre: Skipped 1 previous similar message [ 9437.920126] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp126 [ 9440.224124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9440.228405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9440.234703] Lustre: Skipped 1 previous similar message [ 9440.343366] Lustre: server umount lustre-MDT0000 complete [ 9442.734410] LustreError: 463436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786574795 with bad export cookie 18231844526038385958 [ 9442.741835] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9442.744464] LustreError: 463436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9442.956391] Lustre: server umount lustre-MDT0001 complete [ 9449.324223] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9450.690724] Key type lgssc unregistered [ 9450.900460] LNet: 467334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9450.904155] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9450.907721] LNet: Removed LNI 192.168.202.106@tcp [ 9451.936274] LNet: Removed LNI 192.168.202.106@tcp100 [ 9451.940662] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9451.946180] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9451.952617] LNet: Removed LNI 192.168.202.106@tcp101 [ 9452.960147] LNet: Removed LNI 192.168.202.106@tcp102 [ 9452.962281] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9452.967254] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9452.972022] LNet: Removed LNI 192.168.202.106@tcp103 [ 9453.984293] LNet: Removed LNI 192.168.202.106@tcp104 [ 9455.008219] LNet: Removed LNI 192.168.202.106@tcp105 [ 9455.010504] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9455.014683] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9455.020335] LNet: Removed LNI 192.168.202.106@tcp106 [ 9456.032219] LNet: Removed LNI 192.168.202.106@tcp107 [ 9457.056668] LNet: Removed LNI 192.168.202.106@tcp108 [ 9458.080222] LNet: Removed LNI 192.168.202.106@tcp109 [ 9459.104226] LNet: Removed LNI 192.168.202.106@tcp110 [ 9459.108782] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9459.114469] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9459.124128] LNet: Removed LNI 192.168.202.106@tcp111 [ 9460.128640] LNet: Removed LNI 192.168.202.106@tcp112 [ 9461.152886] LNet: Removed LNI 192.168.202.106@tcp113 [ 9462.176212] LNet: Removed LNI 192.168.202.106@tcp114 [ 9463.200258] LNet: Removed LNI 192.168.202.106@tcp115 [ 9464.224910] LNet: Removed LNI 192.168.202.106@tcp116 [ 9465.248178] LNet: Removed LNI 192.168.202.106@tcp117 [ 9466.272273] LNet: Removed LNI 192.168.202.106@tcp118 [ 9467.296323] LNet: Removed LNI 192.168.202.106@tcp119 [ 9467.298505] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9467.305614] LNetError: 467334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9467.309548] LNet: Removed LNI 192.168.202.106@tcp120 [ 9468.320211] LNet: Removed LNI 192.168.202.106@tcp121 [ 9469.344300] LNet: Removed LNI 192.168.202.106@tcp122 [ 9470.368369] LNet: Removed LNI 192.168.202.106@tcp123 [ 9471.392238] LNet: Removed LNI 192.168.202.106@tcp124 [ 9472.416317] LNet: Removed LNI 192.168.202.106@tcp125 [ 9473.440133] LNet: Removed LNI 192.168.202.106@tcp126 [ 9474.464540] LNet: Removed LNI 192.168.202.106@tcp127 [ 9475.488138] LNet: Removed LNI 192.168.202.106@tcp128 [ 9476.512182] LNet: Removed LNI 192.168.202.106@tcp129 [ 9476.514360] LNet: Removed LNI 192.168.202.106@tcp130 [ 9476.519383] LNet: Removed LNI 192.168.202.106@tcp131 [ 9476.526592] LNet: Removed LNI 192.168.202.106@tcp132 [ 9476.533403] LNet: Removed LNI 192.168.202.106@tcp133 [ 9476.537550] LNet: Removed LNI 192.168.202.106@tcp134 [ 9476.550538] LNet: Removed LNI 192.168.202.106@tcp135 [ 9476.994588] Key type .llcrypt unregistered [ 9476.996273] Key type ._llcrypt unregistered [ 9490.056171] Key type ._llcrypt registered [ 9490.057938] Key type .llcrypt registered [ 9490.142674] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9499.581028] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9500.232230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9500.246717] alg: No test for adler32 (adler32-zlib) [ 9501.152403] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9501.280342] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9502.880350] Key type lgssc registered [ 9503.420310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9521.193849] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:47:52 (1786574872) [ 9530.906645] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9532.534371] Key type lgssc unregistered [ 9532.708466] LNet: 471399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9532.715645] LNetError: 471399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9532.731607] LNet: Removed LNI 192.168.202.106@tcp [ 9533.256643] Key type .llcrypt unregistered [ 9533.258236] Key type ._llcrypt unregistered [ 9542.241679] Key type ._llcrypt registered [ 9542.248150] Key type .llcrypt registered [ 9542.313808] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9542.902311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9542.928208] alg: No test for adler32 (adler32-zlib) [ 9543.803368] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9543.949796] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9545.560672] Key type lgssc registered [ 9546.185667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9550.989984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9551.004509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.108212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9552.128990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9552.182527] Lustre: lustre-MDT0000: new disk, initializing [ 9552.229675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9552.248923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9554.385811] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9560.557661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9560.624613] Lustre: 472589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9560.655857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9560.658076] Lustre: Skipped 1 previous similar message [ 9560.703499] Lustre: lustre-MDT0001: new disk, initializing [ 9560.761236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9560.782065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9560.787340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9563.360470] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9569.139342] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9571.945802] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9575.580812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9575.756301] Lustre: lustre-OST0000: new disk, initializing [ 9575.759895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9575.764320] Lustre: 474554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9575.829856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.262069] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9584.163519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9584.172348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9584.247375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9585.175968] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9587.405060] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing lnet_if_list [ 9590.160342] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9590.175403] LNet: Added LNI 192.168.202.106@tcp10 [8/256/0/180] [ 9593.254500] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9593.275169] LNet: Added LNI 192.168.202.106@tcp11 [8/256/0/180] [ 9595.720785] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9595.739789] LNet: Added LNI 192.168.202.106@tcp12 [8/256/0/180] [ 9598.004290] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9598.019786] LNet: Added LNI 192.168.202.106@tcp13 [8/256/0/180] [ 9600.629884] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9600.648951] LNet: Added LNI 192.168.202.106@tcp14 [8/256/0/180] [ 9603.146995] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9603.165617] LNet: Added LNI 192.168.202.106@tcp15 [8/256/0/180] [ 9606.085402] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9606.102762] LNet: Added LNI 192.168.202.106@tcp16 [8/256/0/180] [ 9608.880994] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9608.900238] LNet: Added LNI 192.168.202.106@tcp17 [8/256/0/180] [ 9611.667054] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9611.680121] LNet: Added LNI 192.168.202.106@tcp18 [8/256/0/180] [ 9614.253966] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9614.275820] LNet: Added LNI 192.168.202.106@tcp19 [8/256/0/180] [ 9617.203806] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9617.218659] LNet: Added LNI 192.168.202.106@tcp20 [8/256/0/180] [ 9620.139441] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9620.162990] LNet: Added LNI 192.168.202.106@tcp21 [8/256/0/180] [ 9623.002147] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9623.014939] LNet: Added LNI 192.168.202.106@tcp22 [8/256/0/180] [ 9626.062545] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9626.079707] LNet: Added LNI 192.168.202.106@tcp23 [8/256/0/180] [ 9629.063715] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9629.082554] LNet: Added LNI 192.168.202.106@tcp24 [8/256/0/180] [ 9632.129338] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9632.144711] LNet: Added LNI 192.168.202.106@tcp25 [8/256/0/180] [ 9635.631618] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9635.650579] LNet: Added LNI 192.168.202.106@tcp26 [8/256/0/180] [ 9639.015604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9639.030663] LNet: Added LNI 192.168.202.106@tcp27 [8/256/0/180] [ 9642.076703] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9642.091070] LNet: Added LNI 192.168.202.106@tcp28 [8/256/0/180] [ 9645.167530] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9645.189246] LNet: Added LNI 192.168.202.106@tcp29 [8/256/0/180] [ 9648.320607] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9648.338550] LNet: Added LNI 192.168.202.106@tcp30 [8/256/0/180] [ 9651.470207] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9651.486761] LNet: Added LNI 192.168.202.106@tcp31 [8/256/0/180] [ 9654.788663] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9654.805244] LNet: Added LNI 192.168.202.106@tcp32 [8/256/0/180] [ 9658.102852] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9658.118744] LNet: Added LNI 192.168.202.106@tcp33 [8/256/0/180] [ 9661.169801] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9661.183412] LNet: Added LNI 192.168.202.106@tcp34 [8/256/0/180] [ 9665.095872] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9665.113115] LNet: Added LNI 192.168.202.106@tcp35 [8/256/0/180] [ 9668.858965] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9668.879912] LNet: Added LNI 192.168.202.106@tcp36 [8/256/0/180] [ 9672.563692] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9672.581962] LNet: Added LNI 192.168.202.106@tcp37 [8/256/0/180] [ 9676.637198] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9676.661203] LNet: Added LNI 192.168.202.106@tcp38 [8/256/0/180] [ 9680.487762] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9680.507765] LNet: Added LNI 192.168.202.106@tcp39 [8/256/0/180] [ 9684.372704] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9684.389187] LNet: Added LNI 192.168.202.106@tcp40 [8/256/0/180] [ 9688.127557] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9688.142618] LNet: Added LNI 192.168.202.106@tcp41 [8/256/0/180] [ 9691.863191] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9691.884423] LNet: Added LNI 192.168.202.106@tcp42 [8/256/0/180] [ 9695.774653] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9695.797428] LNet: Added LNI 192.168.202.106@tcp43 [8/256/0/180] [ 9699.926942] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9699.943962] LNet: Added LNI 192.168.202.106@tcp44 [8/256/0/180] [ 9703.884155] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9703.904242] LNet: Added LNI 192.168.202.106@tcp45 [8/256/0/180] [ 9707.980852] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9708.003787] LNet: Added LNI 192.168.202.106@tcp46 [8/256/0/180] [ 9713.240386] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9713.262851] LNet: Added LNI 192.168.202.106@tcp47 [8/256/0/180] [ 9717.630651] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9717.648662] LNet: Added LNI 192.168.202.106@tcp48 [8/256/0/180] [ 9722.243269] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9722.263175] LNet: Added LNI 192.168.202.106@tcp49 [8/256/0/180] [ 9727.433951] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9727.454350] LNet: Added LNI 192.168.202.106@tcp50 [8/256/0/180] [ 9740.256987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9740.260965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9740.268273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.818256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.821946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.839290] Lustre: Skipped 1 previous similar message [ 9744.779251] Lustre: server umount lustre-MDT0000 complete [ 9747.046296] LustreError: 472580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786575099 with bad export cookie 3745665470539401325 [ 9747.048026] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.060720] LustreError: 472580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9747.250637] Lustre: server umount lustre-MDT0001 complete [ 9758.318328] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9763.296207] Lustre: 471865:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786575100/real 1786575100] req@00000000c036644c x1873359558259072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786575116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9763.315968] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.339971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.416200] Lustre: 471864:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786575105/real 1786575105] req@00000000cd5a688f x1873359558259200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786575121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9771.552658] LustreError: 471861:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7150027 x1873359558259968/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9771.740519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9772.513487] Lustre: 471863:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786575109/real 1786575109] req@000000003fa29233 x1873359558259456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786575125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9774.022437] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9778.858641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.912087] Lustre: 471863:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786575115/real 1786575115] req@00000000a5704cd4 x1873359558259712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786575131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9779.058765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9781.377299] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9783.147797] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9785.096974] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9788.180582] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9798.629820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.634300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9798.639458] Lustre: Skipped 1 previous similar message [ 9798.649323] Lustre: Skipped 3 previous similar messages [ 9803.746225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9803.748973] Lustre: Skipped 1 previous similar message [ 9803.950924] Lustre: server umount lustre-OST0000 complete [ 9808.866450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.869291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.871931] Lustre: Skipped 1 previous similar message [ 9808.877059] Lustre: Skipped 1 previous similar message [ 9819.107327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9819.111395] Lustre: Skipped 3 previous similar messages [ 9820.128140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9820.201490] Lustre: server umount lustre-MDT0000 complete [ 9821.818746] LustreError: 472581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786575174 with bad export cookie 3745665470539403600 [ 9821.821847] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9821.824456] LustreError: 472581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9821.964521] Lustre: server umount lustre-MDT0001 complete [ 9826.977623] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9828.362948] Key type lgssc unregistered [ 9828.564882] LNet: 501529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9828.570616] LNetError: 501529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9828.574805] LNet: Removed LNI 192.168.202.106@tcp [ 9828.576954] LNet: Removed LNI 192.168.202.106@tcp10 [ 9828.578968] LNet: Removed LNI 192.168.202.106@tcp11 [ 9828.581114] LNet: Removed LNI 192.168.202.106@tcp12 [ 9828.583338] LNet: Removed LNI 192.168.202.106@tcp13 [ 9828.585439] LNet: Removed LNI 192.168.202.106@tcp14 [ 9828.587432] LNet: Removed LNI 192.168.202.106@tcp15 [ 9828.589690] LNet: Removed LNI 192.168.202.106@tcp16 [ 9828.591773] LNet: Removed LNI 192.168.202.106@tcp17 [ 9828.593634] LNet: Removed LNI 192.168.202.106@tcp18 [ 9828.594894] LNet: Removed LNI 192.168.202.106@tcp19 [ 9828.596227] LNet: Removed LNI 192.168.202.106@tcp20 [ 9828.597564] LNet: Removed LNI 192.168.202.106@tcp21 [ 9828.598911] LNet: Removed LNI 192.168.202.106@tcp22 [ 9828.601972] LNet: Removed LNI 192.168.202.106@tcp23 [ 9828.604609] LNet: Removed LNI 192.168.202.106@tcp24 [ 9828.607221] LNet: Removed LNI 192.168.202.106@tcp25 [ 9828.609440] LNet: Removed LNI 192.168.202.106@tcp26 [ 9828.611507] LNet: Removed LNI 192.168.202.106@tcp27 [ 9828.613054] LNet: Removed LNI 192.168.202.106@tcp28 [ 9828.614502] LNet: Removed LNI 192.168.202.106@tcp29 [ 9828.616210] LNet: Removed LNI 192.168.202.106@tcp30 [ 9828.618208] LNet: Removed LNI 192.168.202.106@tcp31 [ 9828.620536] LNet: Removed LNI 192.168.202.106@tcp32 [ 9828.623772] LNet: Removed LNI 192.168.202.106@tcp33 [ 9828.625614] LNet: Removed LNI 192.168.202.106@tcp34 [ 9828.628364] LNet: Removed LNI 192.168.202.106@tcp35 [ 9828.630589] LNet: Removed LNI 192.168.202.106@tcp36 [ 9828.632693] LNet: Removed LNI 192.168.202.106@tcp37 [ 9828.634768] LNet: Removed LNI 192.168.202.106@tcp38 [ 9828.637455] LNet: Removed LNI 192.168.202.106@tcp39 [ 9828.639883] LNet: Removed LNI 192.168.202.106@tcp40 [ 9828.645524] LNet: Removed LNI 192.168.202.106@tcp41 [ 9828.648227] LNet: Removed LNI 192.168.202.106@tcp42 [ 9828.650501] LNet: Removed LNI 192.168.202.106@tcp43 [ 9828.652536] LNet: Removed LNI 192.168.202.106@tcp44 [ 9828.655247] LNet: Removed LNI 192.168.202.106@tcp45 [ 9828.657355] LNet: Removed LNI 192.168.202.106@tcp46 [ 9828.659472] LNet: Removed LNI 192.168.202.106@tcp47 [ 9828.661570] LNet: Removed LNI 192.168.202.106@tcp48 [ 9828.662890] LNet: Removed LNI 192.168.202.106@tcp49 [ 9828.668493] LNet: Removed LNI 192.168.202.106@tcp50 [ 9829.096155] Key type .llcrypt unregistered [ 9829.097779] Key type ._llcrypt unregistered [ 9838.639535] Key type ._llcrypt registered [ 9838.640833] Key type .llcrypt registered [ 9838.692743] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9846.858979] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9847.402392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9847.440035] alg: No test for adler32 (adler32-zlib) [ 9848.355609] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9848.496515] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9850.088417] Key type lgssc registered [ 9850.595134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9867.083485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9867.089531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9868.200182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9868.219020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9868.257825] Lustre: lustre-MDT0000: new disk, initializing [ 9868.292134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.301171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9870.345875] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9875.718754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9875.789805] Lustre: 504702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9875.831449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9875.837054] Lustre: Skipped 1 previous similar message [ 9875.883038] Lustre: lustre-MDT0001: new disk, initializing [ 9875.934412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.950432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9875.955802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9878.240878] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9882.960329] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9885.924185] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9890.093028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9890.205381] Lustre: *** cfs_fail_loc=90f, val=0*** [ 9890.251854] Lustre: lustre-OST0000: new disk, initializing [ 9890.254759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9890.259936] Lustre: 506716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9890.309362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9893.129320] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9895.937644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9895.942343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9895.984724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9898.068802] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9906.148456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9906.149458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.157020] LustreError: Skipped 1 previous similar message [ 9906.163269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9906.168969] Lustre: Skipped 1 previous similar message [ 9911.266485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9911.269768] Lustre: Skipped 1 previous similar message [ 9916.384189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9916.388653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9916.400752] Lustre: Skipped 1 previous similar message [ 9916.553846] Lustre: server umount lustre-OST0000 complete [ 9921.509331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.511742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.519266] Lustre: Skipped 2 previous similar messages [ 9921.526680] Lustre: Skipped 1 previous similar message [ 9924.728465] Lustre: server umount lustre-MDT0000 complete [ 9926.496358] LustreError: 504692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786575279 with bad export cookie 7640386536812222499 [ 9926.500551] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9926.501671] LustreError: 504692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9926.645031] Lustre: server umount lustre-MDT0001 complete [ 9931.754126] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9933.184948] Key type lgssc unregistered [ 9933.376963] LNet: 508623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9933.385483] LNetError: 508623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9933.393805] LNet: Removed LNI 192.168.202.106@tcp [ 9933.854374] Key type .llcrypt unregistered [ 9933.857568] Key type ._llcrypt unregistered [ 9944.788866] Key type ._llcrypt registered [ 9944.790465] Key type .llcrypt registered [ 9944.843353] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9945.331170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9945.412230] alg: No test for adler32 (adler32-zlib) [ 9946.338730] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9946.470860] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9948.080537] Key type lgssc registered [ 9948.747307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9952.723334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9952.732036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.967238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9956.042184] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9961.022655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.273178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9963.814167] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9967.952276] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9971.131453] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9974.983618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9975.195383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9978.196185] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9982.239817] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9993.021801] loop0: detected capacity change from 0 to 204800000 [ 9993.168757] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9993.177589] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9993.183697] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9996.753751] loop0: detected capacity change from 0 to 204800000 [ 9996.897280] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9996.904033] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9996.912339] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9999.986082] loop0: detected capacity change from 0 to 204800000 [10002.967116] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10003.041183] Lustre: fs15246-OST0000: new disk, initializing [10003.043925] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10003.048281] Lustre: 513461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10003.083486] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10006.402718] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10012.316649] loop1: detected capacity change from 0 to 204800000 [10015.496538] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.554389] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10015.589919] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10015.640407] Lustre: fs15246-MDT0000: new disk, initializing [10015.730120] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10015.742144] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10015.808052] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10016.866244] Lustre: 514583:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d748e505 x1873359980761216/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786575385 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10017.923242] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10021.924751] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10021.931642] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10021.947178] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10032.101781] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.108220] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10034.471877] Lustre: server umount fs15246-MDT0000 complete [10037.219243] LustreError: 509865:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10037.226563] LustreError: 509865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10038.727587] Lustre: server umount fs15246-OST0000 complete [10044.901821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10044.909773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.919433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10047.461264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10047.465628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.577897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10052.582505] Lustre: Skipped 2 previous similar messages [10057.697462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10057.700437] Lustre: Skipped 1 previous similar message [10058.208567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10058.277692] Lustre: server umount lustre-OST0000 complete [10062.818128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.823019] Lustre: Skipped 1 previous similar message [10067.937222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10067.941325] Lustre: Skipped 3 previous similar messages [10074.080126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10074.160581] Lustre: server umount lustre-MDT0000 complete [10075.854278] LustreError: 509853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786575428 with bad export cookie 14723633267672062966 [10075.856972] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.860974] LustreError: 509853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10075.995426] Lustre: server umount lustre-MDT0001 complete [10080.740583] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10082.045348] Key type lgssc unregistered [10082.211696] LNet: 517678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10082.215382] LNetError: 517678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10082.223402] LNet: Removed LNI 192.168.202.106@tcp [10082.633292] Key type .llcrypt unregistered [10082.638824] Key type ._llcrypt unregistered [10091.581519] Key type ._llcrypt registered [10091.582932] Key type .llcrypt registered [10091.627867] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10091.977581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10092.000454] alg: No test for adler32 (adler32-zlib) [10092.945658] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [10093.080320] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10094.680773] Key type lgssc registered [10095.297100] Lustre: Echo OBD driver; http://www.lustre.org/ [10113.411029] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [10117.202121] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10131.856988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10131.868674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.973110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10132.986258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10133.024534] Lustre: lustre-MDT0000: new disk, initializing [10133.063892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10133.071028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10134.675818] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10139.474035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10139.524297] Lustre: 522115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10139.551821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10139.554270] Lustre: Skipped 1 previous similar message [10139.594092] Lustre: lustre-MDT0001: new disk, initializing [10139.632181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10139.646022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10139.649733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10141.222300] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10145.805889] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10148.187358] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10151.132637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.269441] Lustre: lustre-OST0000: new disk, initializing [10151.272200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10151.276051] Lustre: 524083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10151.335969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10153.849306] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10158.089088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10158.096716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10158.126410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10158.385855] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10160.060898] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10160.143756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10161.812488] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10161.905518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10163.170281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10163.173271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10163.180684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10163.183611] Lustre: Skipped 1 previous similar message [10164.833122] Lustre: server umount lustre-OST0000 complete [10168.289942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10168.291368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.294674] Lustre: Skipped 2 previous similar messages [10172.521556] Lustre: server umount lustre-MDT0000 complete [10174.138514] LustreError: 522108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786575527 with bad export cookie 4028007333307973374 [10174.141907] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10174.143878] LustreError: 522108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10174.273903] Lustre: server umount lustre-MDT0001 complete [10185.083095] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [10189.280381] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10203.387823] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9853 sec ========== 18:59:14 (1786575554) [10204.205749] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:59:15 (1786575555) === [10205.815590] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:59:16 (1786575556) === [10224.153784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10225.490239] Key type lgssc unregistered [10225.652358] LNet: 530717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10225.656747] LNetError: 530717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10225.663354] LNet: Removed LNI 192.168.202.106@tcp [10226.014525] Key type .llcrypt unregistered [10226.015772] Key type ._llcrypt unregistered