[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 395741499 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001006] APIC: Switch to symmetric I/O mode setup [ 0.002141] x2apic enabled [ 0.002932] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005742] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006007] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008068] LSM: Security Framework initializing [ 0.009022] Yama: becoming mindful. [ 0.009633] SELinux: Initializing. [ 0.010048] *** VALIDATE selinux *** [ 0.016632] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.020358] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.021073] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.022059] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023042] *** VALIDATE tmpfs *** [ 0.024305] *** VALIDATE proc *** [ 0.025132] *** VALIDATE cgroup *** [ 0.026004] *** VALIDATE cgroup2 *** [ 0.027110] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028091] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030024] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.033558] debug: unmapping init [mem 0xffffffffae459000-0xffffffffae460fff] [ 0.035684] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036360] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037009] ... version: 2 [ 0.037735] ... bit width: 48 [ 0.038005] ... generic registers: 4 [ 0.038726] ... value mask: 0000ffffffffffff [ 0.039005] ... max period: 00007fffffffffff [ 0.039969] ... fixed-purpose events: 3 [ 0.040005] ... event mask: 000000070000000f [ 0.041067] rcu: Hierarchical SRCU implementation. [ 0.042890] smp: Bringing up secondary CPUs ... [ 0.043300] x86: Booting SMP configuration: [ 0.044010] .... node #0, CPUs: #1 #2 #3 [ 0.046288] smp: Brought up 1 node, 4 CPUs [ 0.047748] smpboot: Max logical packages: 1 [ 0.048006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136519] node 0 deferred pages initialised in 88ms [ 0.139219] devtmpfs: initialized [ 0.140147] x86/mm: Memory block size: 128MB [ 0.142137] gcov: version magic: 0x41383552 [ 0.143450] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144046] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145151] pinctrl core: initialized pinctrl subsystem [ 0.146089] [ 0.146365] ************************************************************* [ 0.147008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148005] ** ** [ 0.149005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150005] ** ** [ 0.151005] ** This means that this kernel is built to expose internal ** [ 0.152005] ** IOMMU data structures, which may compromise security on ** [ 0.153007] ** your system. ** [ 0.154005] ** ** [ 0.155005] ** If you see this message and you are not debugging the ** [ 0.156005] ** kernel, report this immediately to your vendor! ** [ 0.157005] ** ** [ 0.158005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159005] ************************************************************* [ 0.160374] NET: Registered protocol family 16 [ 0.161277] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162024] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163024] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.164232] cpuidle: using governor menu [ 0.165428] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.167422] PCI: Using configuration type 1 for base access [ 0.170159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.180087] cryptd: max_cpu_qlen set to 1000 [ 0.181116] ACPI: Added _OSI(Module Device) [ 0.183019] ACPI: Added _OSI(Processor Device) [ 0.185007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192962] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199358] ACPI: Interpreter enabled [ 0.200052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.201006] ACPI: Using IOAPIC for interrupt routing [ 0.203070] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206410] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216445] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.224028] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229206] acpiphp: Slot [2] registered [ 0.230102] acpiphp: Slot [5] registered [ 0.232095] acpiphp: Slot [6] registered [ 0.233069] acpiphp: Slot [7] registered [ 0.235059] acpiphp: Slot [8] registered [ 0.236078] acpiphp: Slot [9] registered [ 0.237110] acpiphp: Slot [10] registered [ 0.238070] acpiphp: Slot [3] registered [ 0.240041] acpiphp: Slot [4] registered [ 0.241057] acpiphp: Slot [11] registered [ 0.242067] acpiphp: Slot [12] registered [ 0.243059] acpiphp: Slot [13] registered [ 0.244059] acpiphp: Slot [14] registered [ 0.245057] acpiphp: Slot [15] registered [ 0.246070] acpiphp: Slot [16] registered [ 0.247042] acpiphp: Slot [17] registered [ 0.248024] acpiphp: Slot [18] registered [ 0.249010] acpiphp: Slot [19] registered [ 0.250041] acpiphp: Slot [20] registered [ 0.250935] acpiphp: Slot [21] registered [ 0.252054] acpiphp: Slot [22] registered [ 0.252961] acpiphp: Slot [23] registered [ 0.254041] acpiphp: Slot [24] registered [ 0.254789] acpiphp: Slot [25] registered [ 0.256042] acpiphp: Slot [26] registered [ 0.256939] acpiphp: Slot [27] registered [ 0.257041] acpiphp: Slot [28] registered [ 0.257785] acpiphp: Slot [29] registered [ 0.259059] acpiphp: Slot [30] registered [ 0.259857] acpiphp: Slot [31] registered [ 0.261031] PCI host bridge to bus 0000:00 [ 0.261860] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.263009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.264008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.266008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.268018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.271069] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.273169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.276013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279155] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.294465] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.299015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.302008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304374] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307763] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.310021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.312749] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.317006] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.329007] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.332007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.336277] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.346007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.357007] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.365457] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.372008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.377015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.394016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.403000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.409007] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.413010] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.427011] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.433977] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.439007] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.443006] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.454837] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.460000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.463741] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.466800] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.478800] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.485171] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.488767] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.491863] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.503981] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.512281] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.514265] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.516349] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.518277] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.520162] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.524087] iommu: Default domain type: Passthrough [ 0.525261] SCSI subsystem initialized [ 0.527119] ACPI: bus type USB registered [ 0.528067] usbcore: registered new interface driver usbfs [ 0.530033] usbcore: registered new interface driver hub [ 0.531031] usbcore: registered new device driver usb [ 0.532111] pps_core: LinuxPPS API ver. 1 registered [ 0.533005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.535015] PTP clock support registered [ 0.536070] EDAC MC: Ver: 3.0.0 [ 0.538066] PCI: Using ACPI for IRQ routing [ 0.539723] NetLabel: Initializing [ 0.541007] NetLabel: domain hash size = 128 [ 0.542005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.543062] NetLabel: unlabeled traffic allowed by default [ 0.545081] vgaarb: loaded [ 0.546168] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.547008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.555288] clocksource: Switched to clocksource kvm-clock [ 0.636846] VFS: Disk quotas dquot_6.6.0 [ 0.637710] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.639001] *** VALIDATE ramfs *** [ 0.639664] *** VALIDATE hugetlbfs *** [ 0.640382] pnp: PnP ACPI init [ 0.641789] pnp: PnP ACPI: found 6 devices [ 0.653206] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.654891] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.655992] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.657119] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.658325] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.659536] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.660997] NET: Registered protocol family 2 [ 0.662323] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.665229] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.667152] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.670653] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.672446] TCP: Hash tables configured (established 65536 bind 65536) [ 0.673919] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.675488] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.676942] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.678437] NET: Registered protocol family 1 [ 0.679628] RPC: Registered named UNIX socket transport module. [ 0.680714] RPC: Registered udp transport module. [ 0.681549] RPC: Registered tcp transport module. [ 0.682385] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.683530] NET: Registered protocol family 44 [ 0.684358] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.685440] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.686502] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.687693] PCI: CLS 0 bytes, default 64 [ 0.688524] Unpacking initramfs... [ 1.768105] debug: unmapping init [mem 0xffff9bd0bcc54000-0xffff9bd0bffbffff] [ 1.770439] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.771702] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.773259] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.178557] Initialise system trusted keyrings [ 2.179468] Key type blacklist registered [ 2.180337] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.185575] zbud: loaded [ 2.187251] *** VALIDATE nfs *** [ 2.187899] *** VALIDATE nfs4 *** [ 2.188666] pstore: using deflate compression [ 2.190030] Platform Keyring initialized [ 2.244502] NET: Registered protocol family 38 [ 2.245429] Key type asymmetric registered [ 2.246207] Asymmetric key parser 'x509' registered [ 2.247113] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.248543] io scheduler mq-deadline registered [ 2.249405] io scheduler kyber registered [ 2.250227] io scheduler bfq registered [ 2.251133] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.252542] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.254022] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.255397] ACPI: Power Button [PWRF] [ 2.258406] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.262395] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.269488] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.273450] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.281181] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.305765] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.330502] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.333109] Non-volatile memory driver v1.3 [ 2.334034] Linux agpgart interface v0.103 [ 2.352957] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.354707] vda: detected capacity change from 0 to 74825728 [ 2.363622] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.365356] vdb: detected capacity change from 0 to 1073741824 [ 2.374204] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.376012] vdc: detected capacity change from 0 to 2621440000 [ 2.384943] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.386979] vdd: detected capacity change from 0 to 2621440000 [ 2.395947] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.397803] vde: detected capacity change from 0 to 4294967296 [ 2.406827] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.408515] vdf: detected capacity change from 0 to 4294967296 [ 2.411769] libphy: Fixed MDIO Bus: probed [ 2.414720] usbcore: registered new interface driver usbserial_generic [ 2.416183] usbserial: USB Serial support registered for generic [ 2.417488] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.420031] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.420996] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.422367] mousedev: PS/2 mouse device common for all mice [ 2.423891] rtc_cmos 00:05: RTC can wake from S4 [ 2.425539] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.427342] rtc_cmos 00:05: registered as rtc0 [ 2.428839] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.428993] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.430227] intel_pstate: CPU model not supported [ 2.432823] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.435459] hid: raw HID events driver (C) Jiri Kosina [ 2.436513] usbcore: registered new interface driver usbhid [ 2.437518] usbhid: USB HID core driver [ 2.438338] drop_monitor: Initializing network drop monitor service [ 2.439536] Initializing XFRM netlink socket [ 2.440556] NET: Registered protocol family 10 [ 2.442192] Segment Routing with IPv6 [ 2.442944] NET: Registered protocol family 17 [ 2.443996] mpls_gso: MPLS GSO support [ 2.447924] RAS: Correctable Errors collector initialized. [ 2.449014] AVX version of gcm_enc/dec engaged. [ 2.449792] AES CTR mode by8 optimization enabled [ 2.487062] sched_clock: Marking stable (2487049278, 0)->(3072139292, -585090014) [ 2.488776] registered taskstats version 1 [ 2.489734] Loading compiled-in X.509 certificates [ 2.490753] zswap: loaded using pool lzo/zbud [ 2.503674] Key type big_key registered [ 2.509879] Key type encrypted registered [ 2.510737] ima: No TPM chip found, activating TPM-bypass! [ 2.511699] ima: Allocated hash algorithm: sha1 [ 2.512479] ima: No architecture policies found [ 2.513312] evm: Initialising EVM extended attributes: [ 2.514268] evm: security.selinux [ 2.514877] evm: security.ima [ 2.515396] evm: security.capability [ 2.516083] evm: HMAC attrs: 0x1 [ 2.517307] rtc_cmos 00:05: setting system clock to 2026-08-23 14:30:58 UTC (1787495458) [ 2.520860] debug: unmapping init [mem 0xffffffffaf403000-0xffffffffaf5fffff] [ 2.522502] debug: unmapping init [mem 0xffffffffae182000-0xffffffffae458fff] [ 2.533045] Write protecting the kernel read-only data: 28672k [ 2.534753] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 2.536216] debug: unmapping init [mem 0xffffffffad114000-0xffffffffad1fffff] [ 2.555808] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.560017] systemd[1]: Detected virtualization kvm. [ 2.560896] systemd[1]: Detected architecture x86-64. [ 2.561953] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.583088] systemd[1]: No hostname configured. [ 2.583981] systemd[1]: Set hostname to . [ 2.584960] random: systemd: uninitialized urandom read (16 bytes read) [ 2.586194] systemd[1]: Initializing machine ID from random generator. [ 2.604189] random: ln: uninitialized urandom read (6 bytes read) [ 2.665645] random: systemd: uninitialized urandom read (16 bytes read) [ 2.666897] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 2.669238] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 2.671738] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ 2.764670] random: fast init done [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. Starting dracut cmdline hook... [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.042889] device-mapper: uevent: version 1.0.3 [ 3.044077] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 3.478698] virtio_net virtio0 ens2: renamed from eth0 [ 3.500518] scsi host0: ata_piix [ 3.528547] scsi host1: ata_piix [ 3.529536] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.530873] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.511820] dracut-initqueue[591]: RTNETLINK answers: File exists [ 8.028817] random: crng init done [ 8.029676] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 8.245749] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.097218] printk: systemd: 25 output lines suppressed due to ratelimiting [ 9.228585] SELinux: Disabled at runtime. [ 9.275899] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.279993] systemd[1]: Detected virtualization kvm. [ 9.280903] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 9.559229] systemd[1]: initrd-switch-root.service: Succeeded. [ 9.560949] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 9.563612] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 9.565538] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 9.568838] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 9.572708] systemd[1]: Starting Journal Service... Starting Journal Service... [ 9.575158] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Mountin[ 9.651652] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 9.869793] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 10.119034] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.133883] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.263812] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.269048] EDAC sbridge: Ver: 1.1.2 [ 11.381730] Key type dns_resolver registered [ 11.635742] NFS: Registering the id_resolver key type [ 11.636832] Key type id_resolver registered [ 11.637655] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg608-server login: [ 38.336625] libcfs: loading out-of-tree module taints kernel. [ 38.366147] Key type ._llcrypt registered [ 38.367476] Key type .llcrypt registered [ 38.420701] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 47.165677] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 47.797130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 47.803152] alg: No test for adler32 (adler32-zlib) [ 48.834447] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 49.163247] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 50.815150] Key type lgssc registered [ 51.697878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.465675] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 78.391735] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 84.563093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.576358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.703834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.718914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.759029] Lustre: lustre-MDT0000: new disk, initializing [ 85.788510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.796337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 87.640975] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.586858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.654038] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 94.674982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 94.678793] Lustre: Skipped 1 previous similar message [ 94.729583] Lustre: lustre-MDT0001: new disk, initializing [ 94.759973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.772315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 94.776994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 96.709431] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.543404] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 103.990537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 104.131748] Lustre: lustre-OST0000: new disk, initializing [ 104.135059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 104.139631] Lustre: 8428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 104.170230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.798524] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 112.114300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 112.119580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 112.138233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 113.952514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 114.024399] Lustre: lustre-OST0001: new disk, initializing [ 114.027400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 114.031805] Lustre: 9502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 114.062097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.585392] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.842490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 121.847488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 121.870770] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.539473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.793131] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 131.361064] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing check_logdir /tmp/testlogs/ [ 133.336205] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing yml_node [ 135.497957] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 136.580175] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 137.857194] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 138.647990] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 23 10:33:14 EDT 2026 [ 146.572314] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 147.302360] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 161.247863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 161.252682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 161.260313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 162.783710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 162.784552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.787313] Lustre: Skipped 1 previous similar message [ 162.794630] Lustre: Skipped 1 previous similar message [ 165.746674] Lustre: server umount lustre-MDT0000 complete [ 167.452748] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787495623 with bad export cookie 18240539745383379003 [ 167.455121] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 167.459509] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 167.903983] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 167.904699] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 167.911514] Lustre: Skipped 2 previous similar messages [ 167.915032] Lustre: Skipped 2 previous similar messages [ 173.023848] LustreError: 6493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 173.023927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.033635] LustreError: 6493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 173.037208] Lustre: Skipped 1 previous similar message [ 173.696279] Lustre: server umount lustre-MDT0001 complete [ 175.532229] Lustre: server umount lustre-OST0000 complete [ 177.282390] Lustre: server umount lustre-OST0001 complete [ 180.302207] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 184.713260] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 202.050400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.180337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.195959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.237509] Lustre: lustre-MDT0000: new disk, initializing [ 202.264886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.272719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.187603] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 209.628600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.699703] Lustre: 15041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.871891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.732484] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 216.643943] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 219.151800] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 222.392821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.566148] Lustre: lustre-OST0000: new disk, initializing [ 222.568344] Lustre: Skipped 1 previous similar message [ 222.570805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.573916] Lustre: Skipped 2 previous similar messages [ 222.577181] Lustre: 17010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.627347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.630951] Lustre: Skipped 1 previous similar message [ 224.579771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.584893] Lustre: Skipped 1 previous similar message [ 224.588380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.637948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.320617] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 230.205934] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 232.217183] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 232.338751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 234.304793] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 234.433788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 240.101890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 240.103736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.106432] LustreError: Skipped 1 previous similar message [ 240.114757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 240.119689] Lustre: Skipped 1 previous similar message [ 241.518331] Lustre: server umount lustre-OST0000 complete [ 245.219407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.226715] Lustre: Skipped 2 previous similar messages [ 249.526703] Lustre: server umount lustre-MDT0000 complete [ 250.337477] LustreError: 15052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.346826] LustreError: 15052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 251.338388] LustreError: 15033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787495707 with bad export cookie 18240539745383380536 [ 251.345724] LustreError: 15033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 251.347110] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 251.503761] Lustre: server umount lustre-MDT0001 complete [ 253.480066] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:35:08 (1787495708) [ 254.648933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 255.475589] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:35:10 (1787495710) [ 258.603905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.829269] LustreError: 19414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.875414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.874297] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 264.162361] LustreError: 19415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.986141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 267.122393] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 270.588763] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 273.190737] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 276.490654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 279.491361] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 282.899468] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 296.418336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 296.423040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.431728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 296.435753] Lustre: Skipped 3 previous similar messages [ 297.386168] Lustre: server umount lustre-OST0000 complete [ 301.024394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.345124] Lustre: server umount lustre-MDT0000 complete [ 306.658396] LustreError: 19414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.667734] LustreError: 19414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 307.145252] LustreError: 19397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787495763 with bad export cookie 18240539745383381418 [ 307.151356] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.152729] LustreError: 19397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 307.315582] Lustre: server umount lustre-MDT0001 complete [ 312.012656] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 313.546533] Key type lgssc unregistered [ 313.725605] LNet: 22966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.730324] LNetError: 22966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.740561] LNet: Removed LNI 192.168.206.108@tcp [ 314.266172] Key type .llcrypt unregistered [ 314.268265] Key type ._llcrypt unregistered [ 325.909573] Key type ._llcrypt registered [ 325.911267] Key type .llcrypt registered [ 325.971529] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 326.337400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 326.364061] alg: No test for adler32 (adler32-zlib) [ 327.256158] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 327.367578] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 328.959168] Key type lgssc registered [ 329.543158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 333.548705] loop: module loaded [ 333.580750] loop0: detected capacity change from 0 to 204800000 [ 333.666123] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.670376] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.831232] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.747138] loop0: detected capacity change from 0 to 204800000 [ 336.834449] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.838848] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 336.844220] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.314440] loop0: detected capacity change from 0 to 204800000 [ 341.863048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 341.872863] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.985160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 343.005402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 343.059076] Lustre: lustre-MDT0000: new disk, initializing [ 343.088533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.098907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 345.034445] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 349.834903] loop1: detected capacity change from 0 to 204800000 [ 352.488985] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 352.647829] Lustre: lustre-OST07c6: new disk, initializing [ 352.651298] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 352.654876] Lustre: Skipped 1 previous similar message [ 352.658505] Lustre: 26132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 352.701790] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 355.513327] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 356.131536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 356.136884] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 356.155385] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 358.144086] Lustre: 25120:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 358.149643] Lustre: 25120:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 358.154084] Lustre: 25120:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 358.158764] Lustre: 25120:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 358.163573] Lustre: 25120:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 358.168060] Lustre: 25120:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 370.969409] Lustre: 26140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 370.975160] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 370.979654] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 370.984318] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 370.989262] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 370.993707] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 374.741042] Lustre: 25120:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 374.746717] Lustre: 25120:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 374.751077] Lustre: 25120:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 374.755411] Lustre: 25120:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 374.759934] Lustre: 25120:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 374.764444] Lustre: 25120:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 374.768963] Lustre: 25120:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 374.773800] Lustre: 25120:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 374.778268] Lustre: 25120:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 374.782608] Lustre: 25120:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 374.787038] Lustre: 25120:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 374.791353] Lustre: 25120:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.868627] Lustre: 26140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 377.875172] Lustre: 26140:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 377.880395] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 377.885625] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.890968] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 377.896377] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.901743] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 377.907588] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.912917] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 377.918068] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 377.923473] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 377.928687] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 384.677870] Lustre: 26140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 384.683561] Lustre: 26140:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 384.688144] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 384.692627] Lustre: 26140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.697224] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 384.702057] Lustre: 26140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.706705] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 384.711739] Lustre: 26140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.716240] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 384.720746] Lustre: 26140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.725195] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 384.729445] Lustre: 26140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 394.538461] Lustre: 25120:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 394.543930] Lustre: 25120:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 394.548290] Lustre: 25120:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 394.552678] Lustre: 25120:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.557053] Lustre: 25120:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 394.561619] Lustre: 25120:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.565934] Lustre: 25120:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 394.570573] Lustre: 25120:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.574864] Lustre: 25120:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 394.578920] Lustre: 25120:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.583176] Lustre: 25120:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 394.587545] Lustre: 25120:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 438.242492] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.242938] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.255190] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 442.212144] Lustre: server umount lustre-OST07c6 complete [ 446.628203] Lustre: server umount lustre-MDT0000 complete [ 453.823298] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 455.289348] Key type lgssc unregistered [ 455.461668] LNet: 28712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.466483] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 455.477543] LNet: Removed LNI 192.168.206.108@tcp [ 455.928714] Key type .llcrypt unregistered [ 455.931268] Key type ._llcrypt unregistered [ 465.314286] Key type ._llcrypt registered [ 465.316246] Key type .llcrypt registered [ 465.381102] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 465.662159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 465.684194] alg: No test for adler32 (adler32-zlib) [ 466.549693] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 466.644744] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 468.223159] Key type lgssc registered [ 468.830666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 472.718641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 472.730643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.955576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.826241] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 479.990168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.169252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.102420] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 485.598625] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.335973] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.627375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 491.807352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.500826] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 497.962463] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.991302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 514.528439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.533206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.541452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.559785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.560301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.564616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.203499] Lustre: server umount lustre-OST0000 complete [ 520.675720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.676981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.683591] Lustre: Skipped 1 previous similar message [ 520.687398] Lustre: Skipped 1 previous similar message [ 524.283559] Lustre: server umount lustre-MDT0000 complete [ 525.793512] LustreError: 30710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.803172] LustreError: 30710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 526.063967] LustreError: 30703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787495982 with bad export cookie 11802445705074811822 [ 526.074626] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.255715] Lustre: server umount lustre-MDT0001 complete [ 531.406055] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 533.006536] Key type lgssc unregistered [ 533.185862] LNet: 33728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.190572] LNetError: 33728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.200621] LNet: Removed LNI 192.168.206.108@tcp [ 533.696383] Key type .llcrypt unregistered [ 533.698611] Key type ._llcrypt unregistered [ 543.298524] Key type ._llcrypt registered [ 543.300062] Key type .llcrypt registered [ 543.360148] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 543.690381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.714723] alg: No test for adler32 (adler32-zlib) [ 544.597941] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 544.709072] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 546.311147] Key type lgssc registered [ 546.933356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 550.733183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 550.743683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.938833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.800935] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 557.945073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.149778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 560.124809] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 563.597941] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 566.322268] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 569.564080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.714116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.391889] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 575.873159] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 576.877148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 592.353094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 592.357483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.365544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.402882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.403951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.405232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.262332] Lustre: server umount lustre-OST0000 complete [ 599.008396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 599.013052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.021272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 599.024959] Lustre: Skipped 1 previous similar message [ 604.640948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 604.644637] Lustre: Skipped 2 previous similar messages [ 609.761201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 609.764859] Lustre: Skipped 1 previous similar message [ 612.319264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.406438] Lustre: server umount lustre-MDT0000 complete [ 614.131106] LustreError: 34956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496070 with bad export cookie 1897854727325652194 [ 614.137300] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.138048] LustreError: 34956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 614.294443] Lustre: server umount lustre-MDT0001 complete [ 619.037406] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 620.588221] Key type lgssc unregistered [ 620.766590] LNet: 38529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.771377] LNetError: 38529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.783538] LNet: Removed LNI 192.168.206.108@tcp [ 621.220168] Key type .llcrypt unregistered [ 621.222403] Key type ._llcrypt unregistered [ 633.140061] Key type ._llcrypt registered [ 633.141750] Key type .llcrypt registered [ 633.215687] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 633.588924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.605563] alg: No test for adler32 (adler32-zlib) [ 634.515381] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 634.640189] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 636.247175] Key type lgssc registered [ 636.847654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 640.740679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 640.752154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.980929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 643.888296] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 648.065517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.265547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 650.198119] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 653.738190] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 656.492681] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 659.813243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 659.994811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.860106] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 666.159140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 666.503229] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 683.149962] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 683.493557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 683.500919] Lustre: Skipped 2 previous similar messages [ 685.561770] Lustre: server umount lustre-MDT0000 complete [ 687.387303] LustreError: 41765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496143 with bad export cookie 16267092870322694716 [ 687.394456] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 687.394700] LustreError: 41765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 688.255367] LustreError: 39917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 688.265445] LustreError: 39917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 688.271187] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.8@tcp (stopping) [ 688.275257] Lustre: Skipped 3 previous similar messages [ 688.610342] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.640559] LustreError: 40651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.640942] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 692.650605] LustreError: 40651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 692.654375] Lustre: Skipped 1 previous similar message [ 693.599505] Lustre: server umount lustre-MDT0001 complete [ 720.351135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 720.440117] Lustre: server umount lustre-OST0000 complete [ 728.333128] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 729.916974] Key type lgssc unregistered [ 730.101767] LNet: 43810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.106673] LNetError: 43810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.119669] LNet: Removed LNI 192.168.206.108@tcp [ 730.621820] Key type .llcrypt unregistered [ 730.624068] Key type ._llcrypt unregistered [ 742.780903] Key type ._llcrypt registered [ 742.782712] Key type .llcrypt registered [ 742.835875] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 743.343577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 743.353271] alg: No test for adler32 (adler32-zlib) [ 744.245994] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 744.376397] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 745.999155] Key type lgssc registered [ 746.635081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 750.465153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.475869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.675196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.409726] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 757.475910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.667772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 759.289248] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 762.683294] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 765.376247] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 768.554169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.713614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.378942] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 774.934777] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 776.944685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 791.695794] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 792.551244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.555606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.559775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.563175] LustreError: Skipped 1 previous similar message [ 792.566803] Lustre: Skipped 1 previous similar message [ 796.800963] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 801.921232] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 801.925612] Lustre: Skipped 2 previous similar messages [ 802.271162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 802.356575] Lustre: server umount lustre-OST0000 complete [ 816.181615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.182268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 816.189173] Lustre: Skipped 2 previous similar messages [ 822.383352] Lustre: server umount lustre-MDT0000 complete [ 824.111827] LustreError: 45185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496280 with bad export cookie 14210093566943541414 [ 824.117143] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 824.118939] LustreError: 45185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 830.350872] Lustre: server umount lustre-MDT0001 complete [ 835.146079] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 836.654525] Key type lgssc unregistered [ 836.832644] LNet: 48941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.837493] LNetError: 48941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.850655] LNet: Removed LNI 192.168.206.108@tcp [ 837.298348] Key type .llcrypt unregistered [ 837.300495] Key type ._llcrypt unregistered [ 848.451804] Key type ._llcrypt registered [ 848.454014] Key type .llcrypt registered [ 848.519303] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 848.953918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.966086] alg: No test for adler32 (adler32-zlib) [ 849.872251] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 849.985852] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 851.591170] Key type lgssc registered [ 852.176441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.080165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.091638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.313226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.270271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 863.328949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.518655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.238253] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 868.516063] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 871.082777] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 874.257955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.443419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.961393] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 880.317584] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 882.667646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 893.059961] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 894.664124] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 905.620129] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2310h46m00s [ 906.359410] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 907.402327] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 908.414106] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 909.283671] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.290758] Lustre: Skipped 2 previous similar messages [ 910.305676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.309378] Lustre: Skipped 4 previous similar messages [ 913.522372] Lustre: server umount lustre-MDT0000 complete [ 914.557812] LustreError: 50359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.567400] LustreError: 50359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 915.111385] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 916.449798] LustreError: 50353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 916.465605] LustreError: 50353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 918.197176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.269393] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.415458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.460288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 919.462214] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 920.075202] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 921.612126] Lustre: DEBUG MARKER: Wait for df (36622) ... [ 922.350530] Lustre: DEBUG MARKER: done [ 923.617310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.621649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.629607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.646297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.646719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 929.761735] Lustre: server umount lustre-OST0000 complete [ 934.882562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.882942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 934.883771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.883779] Lustre: Skipped 2 previous similar messages [ 934.890055] Lustre: Skipped 1 previous similar message [ 937.707851] Lustre: server umount lustre-MDT0000 complete [ 939.425269] LustreError: 52175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496395 with bad export cookie 13286194722644822891 [ 939.432191] LustreError: 52175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 939.432479] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.577118] Lustre: server umount lustre-MDT0001 complete [ 944.276345] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 945.761980] Key type lgssc unregistered [ 945.926638] LNet: 55275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.931394] LNetError: 55275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.944501] LNet: Removed LNI 192.168.206.108@tcp [ 946.405430] Key type .llcrypt unregistered [ 946.407602] Key type ._llcrypt unregistered [ 965.530717] Key type ._llcrypt registered [ 965.531916] Key type .llcrypt registered [ 965.577368] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 965.895773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 965.933441] alg: No test for adler32 (adler32-zlib) [ 966.826528] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 966.945354] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 968.551162] Key type lgssc registered [ 969.179887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 973.171124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 973.182652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.322375] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 974.335800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 974.428568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.359072] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 980.689187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.748317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 980.767391] Lustre: 57460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 980.881723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 980.938155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.936619] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 986.473220] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 989.064721] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 992.285456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 992.398400] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 992.514158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.791739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 995.187329] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 998.684443] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1002.607342] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2310h47m38s [ 1003.353420] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1006.731507] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1007.910484] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1007.933975] LustreError: 57468:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1028.247586] Lustre: lustre-MDT0000: Client d1f44828-1d1d-4317-9aa4-f35a2dbdc32f (at 192.168.206.8@tcp) reconnecting [ 1047.991161] LustreError: 57468:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1049.029649] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1050.185338] Lustre: DEBUG MARKER: done [ 1053.160137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.162207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1053.168081] Lustre: Skipped 1 previous similar message [ 1058.279848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.283314] Lustre: Skipped 2 previous similar messages [ 1058.408269] Lustre: server umount lustre-OST0000 complete [ 1062.880317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1062.887442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.899820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.353916] Lustre: server umount lustre-MDT0000 complete [ 1068.150229] LustreError: 59320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496524 with bad export cookie 16762351676097757 [ 1068.157048] LustreError: 59320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1068.159843] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1068.319963] Lustre: server umount lustre-MDT0001 complete [ 1073.059169] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1074.668981] Key type lgssc unregistered [ 1074.855807] LNet: 61786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.860777] LNetError: 61786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.872637] LNet: Removed LNI 192.168.206.108@tcp [ 1075.396259] Key type .llcrypt unregistered [ 1075.398505] Key type ._llcrypt unregistered [ 1096.412172] Key type ._llcrypt registered [ 1096.413720] Key type .llcrypt registered [ 1096.480440] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1096.881286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1096.910522] alg: No test for adler32 (adler32-zlib) [ 1097.778304] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1097.878545] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1099.463146] Key type lgssc registered [ 1100.047430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.589830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.599938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.738410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1102.749371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1102.829071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.305340] Lustre: Failing over lustre-MDT0000 [ 1103.460791] Lustre: server umount lustre-MDT0000 complete [ 1106.522194] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:49:21 (1787496561) [ 1109.589252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.796555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.695664] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1115.736060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.787554] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1115.802276] Lustre: 64559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1115.926517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1117.887267] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1121.296379] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1123.862560] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1127.105027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1127.215454] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1127.340705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.081318] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1133.575517] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.709898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1142.067957] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1146.854592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.857182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1146.857796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1146.862577] Lustre: Skipped 1 previous similar message [ 1150.595166] Lustre: server umount lustre-MDT0000 complete [ 1152.416345] LustreError: 64553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496608 with bad export cookie 9722051472203525667 [ 1152.423411] LustreError: 64553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1152.424326] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.581096] Lustre: server umount lustre-MDT0001 complete [ 1154.213659] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1156.414706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1160.478545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.208847] LustreError: 63484:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004b446957 x1874325985592704/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1162.395788] LustreError: 66409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.406985] LustreError: 66409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1162.464376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.523658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1164.503213] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1167.844825] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.845375] LustreError: 66410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.849893] Lustre: Skipped 1 previous similar message [ 1168.641373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.797736] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1173.983193] Lustre: 63486:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787496613/real 1787496613] req@000000007f8c67d5 x1874325985592448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787496629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1173.999856] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1174.173291] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1176.798168] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1177.066164] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1178.079209] Lustre: 63488:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787496618/real 1787496618] req@00000000a30e5a13 x1874325985592576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787496634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1182.179524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.182128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1182.186393] Lustre: Skipped 1 previous similar message [ 1182.190213] Lustre: Skipped 2 previous similar messages [ 1185.390251] Lustre: server umount lustre-MDT0000 complete [ 1187.110252] LustreError: 64554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496643 with bad export cookie 9722051472203527935 [ 1187.116058] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.117163] LustreError: 64554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1187.253440] Lustre: server umount lustre-MDT0001 complete [ 1190.257165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1194.634454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.536451] LustreError: 63484:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006b31bec0 x1874325985613056/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1197.714902] LustreError: 69060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.721937] LustreError: 69060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1197.746261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.748676] Lustre: Skipped 1 previous similar message [ 1197.778714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1199.522909] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1202.853320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.854387] LustreError: 66410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.857316] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1202.861089] Lustre: Skipped 1 previous similar message [ 1202.876819] LustreError: 66410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1203.424471] Lustre: 63488:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787496643/real 1787496643] req@000000009fda6db0 x1874325985612672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787496659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1203.475804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.537454] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1208.810824] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1208.973199] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1209.295074] Lustre: 63485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787496648/real 1787496648] req@0000000012182f1a x1874325985612800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787496664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1211.648627] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.903173] Lustre: 63486:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787496653/real 1787496653] req@000000007a854479 x1874325985612928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787496669 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1219.045538] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.046701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1219.048074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.053089] Lustre: Skipped 1 previous similar message [ 1220.206788] Lustre: server umount lustre-MDT0000 complete [ 1221.957785] LustreError: 64554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496677 with bad export cookie 9722051472203529314 [ 1221.964213] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.964967] LustreError: 64554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1222.112891] Lustre: server umount lustre-MDT0001 complete [ 1225.120642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1226.792377] Lustre: DEBUG MARKER: files compared the same [ 1238.078670] Lustre: server umount lustre-OST0000 complete [ 1245.519071] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1247.130813] Key type lgssc unregistered [ 1247.307679] LNet: 74283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.312502] LNetError: 74283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.321613] LNet: Removed LNI 192.168.206.108@tcp [ 1247.804224] Key type .llcrypt unregistered [ 1247.806288] Key type ._llcrypt unregistered [ 1259.138820] Key type ._llcrypt registered [ 1259.140191] Key type .llcrypt registered [ 1259.209178] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1259.643995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1259.699183] alg: No test for adler32 (adler32-zlib) [ 1260.598789] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1260.721459] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1262.335151] Key type lgssc registered [ 1262.938063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1266.879451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.891033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.128754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.069587] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug +malloc all [ 1274.092718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.246839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.983089] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug +malloc all [ 1279.357833] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.873867] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1285.051777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1285.194523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.769534] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug +malloc all [ 1291.170747] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.411708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1309.151906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.156404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.163938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.177242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.177325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.190678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.296370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.299956] Lustre: Skipped 2 previous similar messages [ 1319.391162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.463806] Lustre: server umount lustre-OST0000 complete [ 1325.536218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.536301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.536813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.543604] Lustre: Skipped 1 previous similar message [ 1327.404729] Lustre: server umount lustre-MDT0000 complete [ 1329.141260] LustreError: 75657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496785 with bad export cookie 9087095815218360631 [ 1329.144355] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1329.148581] LustreError: 75657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1329.291365] Lustre: server umount lustre-MDT0001 complete [ 1334.115867] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1335.592208] Key type lgssc unregistered [ 1335.770472] LNet: 79232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.775076] LNetError: 79232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.783532] LNet: Removed LNI 192.168.206.108@tcp [ 1336.230382] Key type .llcrypt unregistered [ 1336.233590] Key type ._llcrypt unregistered [ 1347.629597] Key type ._llcrypt registered [ 1347.630767] Key type .llcrypt registered [ 1347.687534] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1348.045588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.080674] alg: No test for adler32 (adler32-zlib) [ 1348.968551] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1349.090768] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1350.687160] Key type lgssc registered [ 1351.243070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1354.889358] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1354.900251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.399400] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1375.519306] LustreError: 80599:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [ 1402.143231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1404.696873] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1408.194290] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1411.914328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.982779] Lustre: *** cfs_fail_loc=706, val=0*** [ 1412.201311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.246506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1413.972765] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1418.112626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.328108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.380406] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1423.920775] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1426.518303] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1428.452425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1428.454933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.460932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.464689] Lustre: Skipped 1 previous similar message [ 1433.577892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.581839] Lustre: Skipped 2 previous similar messages [ 1433.659477] Lustre: server umount lustre-OST0000 complete [ 1438.692127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.692780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1438.701734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.707995] Lustre: Skipped 1 previous similar message [ 1441.658773] Lustre: server umount lustre-MDT0000 complete [ 1443.465245] LustreError: 80607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787496899 with bad export cookie 16612982711937585174 [ 1443.472082] LustreError: 80607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1443.474778] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1443.628977] Lustre: server umount lustre-MDT0001 complete [ 1448.434529] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1450.034554] Key type lgssc unregistered [ 1450.215867] LNet: 84217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.220920] LNetError: 84217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.235628] LNet: Removed LNI 192.168.206.108@tcp [ 1450.747372] Key type .llcrypt unregistered [ 1450.749517] Key type ._llcrypt unregistered [ 1462.924655] Key type ._llcrypt registered [ 1462.925997] Key type .llcrypt registered [ 1462.977277] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1463.370158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1463.443743] alg: No test for adler32 (adler32-zlib) [ 1464.315615] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1464.416741] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1466.031166] Key type lgssc registered [ 1466.669238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.609575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.621070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.623204] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1477.350951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.584413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.552958] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1482.721209] LustreError: 86297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.730598] LustreError: 86297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1483.706619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.879230] LustreError: 86295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.886891] LustreError: 86295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1483.893752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.717297] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1487.969460] LustreError: 86301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.979401] LustreError: 86301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1490.895882] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.904708] Lustre: MGS: Client 4c40b08e-7a3d-4c72-ae55-30137c28becb (at 0@lo) reconnecting [ 1490.923889] LustreError: 86311:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1491.050579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.102146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1492.924194] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1501.668116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.676609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1506.273908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.278909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.287657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.689711] Lustre: server umount lustre-OST0000 complete [ 1511.907330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1511.909065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.910652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.910661] Lustre: Skipped 1 previous similar message [ 1511.925297] Lustre: Skipped 1 previous similar message [ 1515.674952] Lustre: server umount lustre-MDT0000 complete [ 1517.027193] LustreError: 87841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.036724] LustreError: 87841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1517.651827] Lustre: server umount lustre-MDT0001 complete [ 1519.478314] Lustre: server umount MGS complete [ 1524.588392] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:56:20 (1787496980) [ 1537.657378] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 1542.100651] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1559.236108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.227590] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1566.401730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.523940] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1566.546134] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1566.598074] Lustre: lustre-MDT0001: new disk, initializing [ 1566.638846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.653692] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1566.658036] Lustre: 93461:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1566.663232] LustreError: 93461:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1568.623769] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1574.169913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.330558] Lustre: lustre-OST0000: new disk, initializing [ 1574.333659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1574.338240] Lustre: 94228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1574.381071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.223405] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1580.064747] Lustre: 94684:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000022c62e7f x1874326370421248/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787497052 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1582.241938] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1585.214189] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1585.222510] Lustre: MGS: Client 73e30539-6df9-47b6-9c54-afc81eef9823 (at 0@lo) reconnecting [ 1585.251595] Lustre: 95275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1585.269200] LustreError: 93446:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1585.274476] LustreError: 93446:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1585.284146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1585.336410] Lustre: lustre-MDT0000: new disk, initializing [ 1585.416628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.431342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1586.463822] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1586.468316] Lustre: 95300:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000599245b2 x1874326370427136/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787497058 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1587.402082] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1594.339735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1594.344743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1595.555547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1595.561200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1595.621783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1618.406465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.410847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.413821] Lustre: Skipped 1 previous similar message [ 1618.417831] Lustre: Skipped 1 previous similar message [ 1623.279573] Lustre: server umount lustre-OST0000 complete [ 1628.642980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1628.644321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.657595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.661300] Lustre: Skipped 1 previous similar message [ 1631.390336] Lustre: server umount lustre-MDT0000 complete [ 1639.400774] Lustre: server umount lustre-MDT0001 complete [ 1641.238838] Lustre: server umount MGS complete [ 1646.531097] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:58:21 (1787497101) [ 1654.634904] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1656.214237] Key type lgssc unregistered [ 1656.405621] LNet: 97798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.410251] LNetError: 97798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.419783] LNet: Removed LNI 192.168.206.108@tcp [ 1656.941481] Key type .llcrypt unregistered [ 1656.943604] Key type ._llcrypt unregistered [ 1665.916614] Key type ._llcrypt registered [ 1665.918475] Key type .llcrypt registered [ 1665.983802] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1666.384332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.439439] alg: No test for adler32 (adler32-zlib) [ 1667.344094] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1667.452538] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1669.047143] Key type lgssc registered [ 1669.674924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1673.441150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1673.454236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.672979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.644121] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1680.811510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.015712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1682.906899] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1686.492769] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1689.216215] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1692.481400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1692.676344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.462693] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1699.040778] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1699.887426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1715.169355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1715.173890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.181883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.197208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1716.200971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.201740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.445085] Lustre: server umount lustre-OST0000 complete [ 1721.316298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.317514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.323694] Lustre: Skipped 1 previous similar message [ 1721.327732] Lustre: Skipped 1 previous similar message [ 1726.391487] Lustre: server umount lustre-MDT0000 complete [ 1726.432814] LustreError: 98996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.442447] LustreError: 98996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1728.185929] LustreError: 98979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497184 with bad export cookie 7477463185123842638 [ 1728.192701] LustreError: 98979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1728.193336] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.358520] Lustre: server umount lustre-MDT0001 complete [ 1733.538910] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1735.095349] Key type lgssc unregistered [ 1735.278723] LNet: 102549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.283704] LNetError: 102549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.293746] LNet: Removed LNI 192.168.206.108@tcp [ 1735.767855] Key type .llcrypt unregistered [ 1735.769613] Key type ._llcrypt unregistered [ 1744.317246] Key type ._llcrypt registered [ 1744.318575] Key type .llcrypt registered [ 1744.369826] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1744.723628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1744.730685] alg: No test for adler32 (adler32-zlib) [ 1745.639978] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1745.773752] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1747.375193] Key type lgssc registered [ 1748.007992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1751.025804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1751.025877] LustreError: 103615:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1751.030057] Lustre: Skipped 1 previous similar message [ 1751.036247] LustreError: 103616:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1751.040181] LustreError: 103615:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1751.045174] LustreError: 103615:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1751.049290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.051702] LustreError: 103615:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1752.281571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.994991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.146187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.939417] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1760.283175] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing lsmod [ 1761.735509] LustreError: 104821:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1761.735583] LustreError: 104822:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1761.744085] LustreError: 104821:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1761.748373] LustreError: 104821:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1761.752523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.754772] LustreError: 104821:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1761.946323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.992797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1766.375375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.375645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.382459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.382776] Lustre: Skipped 1 previous similar message [ 1769.461744] Lustre: server umount lustre-MDT0000 complete [ 1771.263316] LustreError: 103679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497227 with bad export cookie 13921434306033978676 [ 1771.271498] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.488654] LustreError: 103677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.488868] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.498791] LustreError: 103677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1771.506223] Lustre: Skipped 1 previous similar message [ 1771.514797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1771.518272] Lustre: Skipped 2 previous similar messages [ 1772.511331] LustreError: 104086:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001af5ed42 x1874326665053568/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1772.597587] Lustre: server umount lustre-MDT0001 complete [ 1774.512215] Lustre: server umount lustre-OST0000 complete [ 1778.627971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.877774] LustreError: 105884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1778.910870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.860342] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1784.289284] LustreError: 105885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.038276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.225123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1787.182324] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1790.770371] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1793.446772] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1796.765463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1796.960987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.694270] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1803.122731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 1803.266613] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1818.592135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1818.596784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.604713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.644412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.645975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1820.653099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.443434] Lustre: server umount lustre-OST0000 complete [ 1825.763139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.764845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.770629] Lustre: Skipped 1 previous similar message [ 1825.774739] Lustre: Skipped 1 previous similar message [ 1830.397118] Lustre: server umount lustre-MDT0000 complete [ 1830.881792] LustreError: 105885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.892797] LustreError: 105885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1832.204200] LustreError: 106619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497288 with bad export cookie 13921434306033979593 [ 1832.211138] LustreError: 106619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1832.213906] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1832.352100] Lustre: server umount lustre-MDT0001 complete [ 1837.596620] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1839.120835] Key type lgssc unregistered [ 1839.297780] LNet: 109454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.302940] LNetError: 109454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.312781] LNet: Removed LNI 192.168.206.108@tcp [ 1839.806940] Key type .llcrypt unregistered [ 1839.809126] Key type ._llcrypt unregistered [ 1852.158584] Key type ._llcrypt registered [ 1852.159790] Key type .llcrypt registered [ 1852.220409] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1852.592558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.604910] alg: No test for adler32 (adler32-zlib) [ 1853.500349] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1853.608828] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1855.215138] Key type lgssc registered [ 1855.826614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.681992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.693192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.887679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.678961] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1866.773669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.951834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.929300] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1872.380632] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1874.958459] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1878.177546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1878.375728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.195352] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1884.676825] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1885.554336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1896.943631] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1897.893772] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1901.025555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.030070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.038058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.052369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.056668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.061167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.253463] Lustre: server umount lustre-OST0000 complete [ 1907.169982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.171141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.177412] Lustre: Skipped 1 previous similar message [ 1907.181053] Lustre: Skipped 1 previous similar message [ 1912.289171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.292832] Lustre: Skipped 1 previous similar message [ 1913.277004] Lustre: server umount lustre-MDT0000 complete [ 1915.075773] LustreError: 110831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497371 with bad export cookie 10298377250448458288 [ 1915.082291] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.082569] LustreError: 110831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1915.236612] Lustre: server umount lustre-MDT0001 complete [ 1920.345944] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 1921.902677] Key type lgssc unregistered [ 1922.073881] LNet: 114506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.079238] LNetError: 114506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.088779] LNet: Removed LNI 192.168.206.108@tcp [ 1922.590905] Key type .llcrypt unregistered [ 1922.593123] Key type ._llcrypt unregistered [ 1931.712351] Key type ._llcrypt registered [ 1931.714157] Key type .llcrypt registered [ 1931.778718] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 1932.296214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.305934] alg: No test for adler32 (adler32-zlib) [ 1933.185182] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 1933.311576] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 1934.919182] Key type lgssc registered [ 1935.538758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1939.467923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1939.478416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.674826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.668992] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1946.711418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.923924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.805574] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1952.357975] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1955.081124] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.446143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.656379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.471339] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 1965.004313] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1965.872903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1981.408415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1981.413245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.421731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.436255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1982.439778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1982.441205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.424699] Lustre: server umount lustre-OST0000 complete [ 1987.555565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.556580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1987.563059] Lustre: Skipped 1 previous similar message [ 1987.566744] Lustre: Skipped 1 previous similar message [ 1992.383426] Lustre: server umount lustre-MDT0000 complete [ 1992.674256] LustreError: 115705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.683471] LustreError: 115705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1994.167815] LustreError: 115685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497450 with bad export cookie 16337415233907128933 [ 1994.174733] LustreError: 115685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1994.176324] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.332955] Lustre: server umount lustre-MDT0001 complete [ 1999.307548] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2000.891205] Key type lgssc unregistered [ 2001.072802] LNet: 119263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.077403] LNetError: 119263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.087740] LNet: Removed LNI 192.168.206.108@tcp [ 2001.566495] Key type .llcrypt unregistered [ 2001.568836] Key type ._llcrypt unregistered [ 2013.452678] Key type ._llcrypt registered [ 2013.454380] Key type .llcrypt registered [ 2013.520461] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2014.042122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2014.078958] alg: No test for adler32 (adler32-zlib) [ 2014.950482] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2015.057578] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2016.647163] Key type lgssc registered [ 2017.258387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2021.224217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2021.234990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.456122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.353659] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2028.489421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.715912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.765006] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2034.363626] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.021051] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2040.282354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2040.485242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.265602] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2046.771518] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2047.660060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 2059.897128] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2061.605155] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2061.610448] Lustre: Skipped 2 previous similar messages [ 2063.567541] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2063.572041] Lustre: Skipped 2 previous similar messages [ 2067.983298] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2068.003372] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2070.695157] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.8@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2070.701442] Lustre: Skipped 2 previous similar messages [ 2076.680402] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.8@tcp 192.168.0.[2,10]@tcp [ 2079.401237] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.8/24@tcp in log lustre-MDT0000 [ 2079.406277] Lustre: Skipped 2 previous similar messages [ 2086.925689] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.8/24@tcp [ 2086.929106] Lustre: Skipped 1 previous similar message [ 2089.954522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.956852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2089.957557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.961943] Lustre: Skipped 1 previous similar message [ 2095.073768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.077575] Lustre: Skipped 1 previous similar message [ 2096.101430] Lustre: server umount lustre-OST0000 complete [ 2100.194505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.195502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.201806] Lustre: Skipped 1 previous similar message [ 2104.047592] Lustre: server umount lustre-MDT0000 complete [ 2105.313935] LustreError: 120675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.323270] LustreError: 120675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2105.764810] LustreError: 120637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497561 with bad export cookie 12361709596152198925 [ 2105.770928] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.771707] LustreError: 120637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2105.932475] Lustre: server umount lustre-MDT0001 complete [ 2111.070570] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2112.607464] Key type lgssc unregistered [ 2112.774713] LNet: 125851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.779346] LNetError: 125851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.790587] LNet: Removed LNI 192.168.206.108@tcp [ 2113.268919] Key type .llcrypt unregistered [ 2113.271133] Key type ._llcrypt unregistered [ 2127.571073] Key type ._llcrypt registered [ 2127.572255] Key type .llcrypt registered [ 2127.616943] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2127.932348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2127.938980] alg: No test for adler32 (adler32-zlib) [ 2128.844402] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2128.956994] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2130.551145] Key type lgssc registered [ 2131.155866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.095276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2135.106449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.270033] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2136.272468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.278318] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.8/24@tcp [ 2138.129922] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2142.185233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.334418] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2142.348528] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.8/24@tcp [ 2142.377607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2144.157644] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2147.669248] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.369923] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2153.646858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.799512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.454237] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2159.981658] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2162.031625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 2177.507160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2177.510333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.513703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2177.518975] Lustre: Skipped 1 previous similar message [ 2180.006423] Lustre: server umount lustre-OST0000 complete [ 2182.626511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.627570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.634037] Lustre: Skipped 1 previous similar message [ 2182.637797] Lustre: Skipped 1 previous similar message [ 2187.745415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.751247] Lustre: Skipped 2 previous similar messages [ 2187.950928] Lustre: server umount lustre-MDT0000 complete [ 2189.718890] LustreError: 129971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497645 with bad export cookie 2067144047002634110 [ 2189.723924] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.725496] LustreError: 129971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2189.877347] Lustre: server umount lustre-MDT0001 complete [ 2194.852486] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2196.337208] Key type lgssc unregistered [ 2196.492651] LNet: 131011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.497068] LNetError: 131011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.505574] LNet: Removed LNI 192.168.206.108@tcp [ 2196.949258] Key type .llcrypt unregistered [ 2196.951097] Key type ._llcrypt unregistered [ 2209.111843] Key type ._llcrypt registered [ 2209.113360] Key type .llcrypt registered [ 2209.174272] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2217.376349] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2217.903544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.916623] alg: No test for adler32 (adler32-zlib) [ 2218.839791] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2218.975480] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2220.583146] Key type lgssc registered [ 2221.149386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2237.928349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2237.941795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.054749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2239.072651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2239.127226] Lustre: lustre-MDT0000: new disk, initializing [ 2239.155280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.166271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2241.041328] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2246.480646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.545859] Lustre: 134385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2246.587834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2246.591517] Lustre: Skipped 1 previous similar message [ 2246.638918] Lustre: lustre-MDT0001: new disk, initializing [ 2246.720221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.742820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2246.751406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2248.619355] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2253.529781] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2256.156173] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.397830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.550310] Lustre: lustre-OST0000: new disk, initializing [ 2259.553281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2259.557733] Lustre: 136356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2259.603347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2262.248028] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2267.083600] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2267.153159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2267.158497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2267.210228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2269.103513] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2269.223272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2271.087166] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2271.206924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2272.227862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2272.228780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.231498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2272.233657] LustreError: Skipped 1 previous similar message [ 2272.242268] Lustre: Skipped 1 previous similar message [ 2274.220343] Lustre: server umount lustre-OST0000 complete [ 2277.346702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2277.347383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.363883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.369445] Lustre: Skipped 2 previous similar messages [ 2282.161120] Lustre: server umount lustre-MDT0000 complete [ 2283.925120] LustreError: 134376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497739 with bad export cookie 837265640030500501 [ 2283.932412] LustreError: 134376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2283.932918] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.053613] Lustre: server umount lustre-MDT0001 complete [ 2288.176326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.422401] LustreError: 138568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.433281] LustreError: 138568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2288.462753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.395400] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2293.731692] LustreError: 138569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.515550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.661782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2296.597845] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2300.215222] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2302.983413] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2306.240333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.435135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.997113] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2312.413664] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2314.329430] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2314.435711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2316.358093] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2320.596234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 2325.087789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.198423] Lustre: lustre-OST0001: new disk, initializing [ 2325.201786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2325.206896] Lustre: 141968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2325.262397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2328.056825] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2332.477894] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2332.705663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2332.713238] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2332.774877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2333.642402] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2335.607422] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2335.740585] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2338.518881] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b1e2020e800.ost_server_uuid 50 [ 2339.328184] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b1e2020e800.ost_server_uuid in FULL state after 0 sec [ 2342.892256] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2342.895223] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.898532] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2342.903927] Lustre: Skipped 2 previous similar messages [ 2348.014585] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2348.018078] Lustre: Skipped 2 previous similar messages [ 2348.113314] Lustre: server umount lustre-OST0001 complete [ 2350.560408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.565020] LustreError: Skipped 1 previous similar message [ 2350.567744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.080618] LustreError: 143366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.089885] LustreError: 143366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2356.198461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2356.202089] Lustre: Skipped 3 previous similar messages [ 2356.203725] LustreError: 140387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.213882] LustreError: 140387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2361.319482] LustreError: 141931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.328849] LustreError: 141931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2364.383183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2364.472188] Lustre: server umount lustre-OST0000 complete [ 2366.434335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.437123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2366.441731] Lustre: Skipped 2 previous similar messages [ 2366.445488] Lustre: Skipped 2 previous similar messages [ 2366.448368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2372.488976] Lustre: server umount lustre-MDT0000 complete [ 2374.285991] LustreError: 138548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497830 with bad export cookie 837265640030501369 [ 2374.293991] LustreError: 138548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2374.295490] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.487710] Lustre: server umount lustre-MDT0001 complete [ 2383.586809] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2385.187915] Key type lgssc unregistered [ 2385.379790] LNet: 144978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.384861] LNetError: 144978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.394612] LNet: Removed LNI 192.168.206.108@tcp [ 2385.890134] Key type .llcrypt unregistered [ 2385.892325] Key type ._llcrypt unregistered [ 2404.447256] Key type ._llcrypt registered [ 2404.448515] Key type .llcrypt registered [ 2404.510063] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2412.623221] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2413.238749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.247811] alg: No test for adler32 (adler32-zlib) [ 2414.133960] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2414.246872] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2415.855208] Key type lgssc registered [ 2416.477212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2433.297841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2433.307559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.426313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2434.439609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2434.483338] Lustre: lustre-MDT0000: new disk, initializing [ 2434.515505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.523875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2436.503546] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2442.061662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.135777] Lustre: 149103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2442.180407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2442.184086] Lustre: Skipped 1 previous similar message [ 2442.229967] Lustre: lustre-MDT0001: new disk, initializing [ 2442.300441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2442.320401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2442.327449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2444.238060] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2449.136279] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2451.574856] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2454.866126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2455.035929] Lustre: lustre-OST0000: new disk, initializing [ 2455.038429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2455.042131] Lustre: 151074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2455.090102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.807978] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2462.626974] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2464.270992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2464.277499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2464.342624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2470.172930] Lustre: Failing over lustre-OST0000 [ 2470.271766] Lustre: server umount lustre-OST0000 complete [ 2484.380267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.579142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.585865] Lustre: Skipped 1 previous similar message [ 2484.592445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.607914] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2487.183818] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2489.020183] Lustre: Failing over lustre-MDT0000 [ 2489.164736] Lustre: server umount lustre-MDT0000 complete [ 2489.485625] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2489.828246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.835616] Lustre: Skipped 2 previous similar messages [ 2490.847222] Lustre: 147331:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787497930/real 1787497930] req@000000001d780b19 x1874327366562816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787497946 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2494.591084] LustreError: 149110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.601689] LustreError: 149110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2495.967258] Lustre: 147330:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787497935/real 1787497935] req@000000003901bdd9 x1874327366564352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787497951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2499.711548] LustreError: 149110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.721550] LustreError: 149110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2503.475747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.537492] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.543598] Lustre: 149107:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2503.671931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2503.704216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.751511] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2504.830970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2505.702401] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2508.786939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2508.799246] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2508.847127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2513.896601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.897946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2513.901076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2513.904270] Lustre: Skipped 1 previous similar message [ 2516.328245] Lustre: server umount lustre-OST0000 complete [ 2519.011383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2519.013079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.016051] LustreError: Skipped 1 previous similar message [ 2519.024324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.029905] Lustre: Skipped 2 previous similar messages [ 2524.129602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.133447] Lustre: Skipped 1 previous similar message [ 2524.355392] Lustre: server umount lustre-MDT0000 complete [ 2526.165207] LustreError: 149095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787497982 with bad export cookie 17843179473276818276 [ 2526.172215] LustreError: 149095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2526.174840] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.354662] Lustre: server umount lustre-MDT0001 complete [ 2531.576973] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 2533.203385] Key type lgssc unregistered [ 2533.382712] LNet: 154724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.387478] LNetError: 154724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.397673] LNet: Removed LNI 192.168.206.108@tcp [ 2533.905142] Key type .llcrypt unregistered [ 2533.907316] Key type ._llcrypt unregistered [ 2557.652951] Key type ._llcrypt registered [ 2557.654095] Key type .llcrypt registered [ 2557.711181] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2558.189139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.228837] alg: No test for adler32 (adler32-zlib) [ 2559.096872] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 2559.212804] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 2560.815158] Key type lgssc registered [ 2561.400046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2565.264066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2565.274819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.466189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.461335] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2573.385734] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2575.836435] LustreError: 156272:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2575.843364] LustreError: 156272:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2575.849239] LustreError: 156272:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2576.025138] Lustre: server umount lustre-MDT0000 complete [ 2583.237728] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2587.935777] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2605.793666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.942661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2605.962045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2606.019925] Lustre: lustre-MDT0000: new disk, initializing [ 2606.054478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.065922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2607.866396] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2613.247407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.320693] Lustre: 159571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2613.361776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2613.365403] Lustre: Skipped 1 previous similar message [ 2613.412720] Lustre: lustre-MDT0001: new disk, initializing [ 2613.456578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2613.477958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2613.484262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2615.235550] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2619.978503] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2622.484216] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2625.621358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.781406] Lustre: lustre-OST0000: new disk, initializing [ 2625.784820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2625.789695] Lustre: 161538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2625.840565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.425945] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2633.225822] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2635.273115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2635.279175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2635.332697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2640.450732] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2641.283327] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:14:56 (1787498096) [ 2737.632163] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.632567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.639621] Lustre: Skipped 2 previous similar messages [ 2737.645927] Lustre: Skipped 2 previous similar messages [ 2742.751851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.755463] Lustre: Skipped 2 previous similar messages [ 2743.101135] Lustre: server umount lustre-MDT0000 complete [ 2744.789982] LustreError: 159564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498200 with bad export cookie 7319248647459216002 [ 2744.791966] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.797239] LustreError: 159564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2747.871670] LustreError: 162063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.871782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.881043] LustreError: 162063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2747.888741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2749.919607] LustreError: 159578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.919613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2755.039625] LustreError: 162063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.039695] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2759.135170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2759.234982] Lustre: server umount lustre-MDT0001 complete [ 2767.199526] Lustre: server umount lustre-OST0000 complete [ 2771.285305] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2775.903206] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2793.191295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.331062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2793.347270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2793.391052] Lustre: lustre-MDT0000: new disk, initializing [ 2793.414504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.421882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2795.257085] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2800.784215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.848633] Lustre: 166172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2800.903957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2800.905089] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2800.912213] Lustre: Skipped 1 previous similar message [ 2800.960405] Lustre: lustre-MDT0001: new disk, initializing [ 2801.021776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2801.029615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2802.810562] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2807.752621] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2810.467655] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2813.726377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2813.887464] Lustre: lustre-OST0000: new disk, initializing [ 2813.890903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2813.895751] Lustre: 168139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2813.945794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.949865] Lustre: Skipped 1 previous similar message [ 2816.377482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2816.380200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2816.415149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2816.512403] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2821.406187] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2823.385278] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2823.500371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2825.335873] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2825.456785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2826.724934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2826.726854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.729447] LustreError: Skipped 1 previous similar message [ 2826.730597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2826.736866] Lustre: Skipped 1 previous similar message [ 2832.486609] Lustre: server umount lustre-OST0000 complete [ 2836.964114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2836.964709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.976305] Lustre: Skipped 1 previous similar message [ 2840.511407] Lustre: server umount lustre-MDT0000 complete [ 2842.080939] LustreError: 166183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.090688] LustreError: 166183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2842.324531] LustreError: 166981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498298 with bad export cookie 7319248647459374818 [ 2842.329483] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.331963] LustreError: 166981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2842.484731] Lustre: server umount lustre-MDT0001 complete [ 2846.673730] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:18:22 (1787498302) [ 2854.087661] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2858.239856] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2875.074948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.191626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2875.205775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2875.254661] Lustre: lustre-MDT0000: new disk, initializing [ 2875.278116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.284821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2877.059399] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2882.578358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.648456] Lustre: 172889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2882.783339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2884.512497] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2889.389369] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2891.897412] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2895.062527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.232047] Lustre: 174867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2897.847342] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2902.607552] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2903.560987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2903.619121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2913.765849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2913.767594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.770471] LustreError: Skipped 1 previous similar message [ 2913.779114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.784291] Lustre: Skipped 6 previous similar messages [ 2916.642328] Lustre: server umount lustre-OST0000 complete [ 2918.882201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2924.591246] Lustre: server umount lustre-MDT0000 complete [ 2926.327117] LustreError: 172882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498382 with bad export cookie 7319248647459375700 [ 2926.333392] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.334194] LustreError: 172882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2926.489380] Lustre: server umount lustre-MDT0001 complete [ 2930.306572] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:19:45 (1787498385) [ 2937.635246] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 2942.147390] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 2959.492310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.639909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2959.658352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2959.662453] Lustre: Skipped 3 previous similar messages [ 2959.717078] Lustre: lustre-MDT0000: new disk, initializing [ 2959.718546] Lustre: Skipped 2 previous similar messages [ 2959.755990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.760189] Lustre: Skipped 2 previous similar messages [ 2959.770049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2959.775390] Lustre: Skipped 2 previous similar messages [ 2961.720332] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2967.330765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.408647] Lustre: 179179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2967.579983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2969.514499] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2974.571941] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2977.242068] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2980.564583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2980.734412] Lustre: 181160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2983.619872] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 2988.584250] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2990.170455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3000.295309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.296335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.299982] LustreError: Skipped 1 previous similar message [ 3000.307476] Lustre: Skipped 3 previous similar messages [ 3000.308861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.316850] Lustre: Skipped 5 previous similar messages [ 3002.994244] Lustre: server umount lustre-OST0000 complete [ 3019.231181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.320974] Lustre: server umount lustre-MDT0000 complete [ 3020.770347] LustreError: 179191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.780440] LustreError: 179191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3021.164116] LustreError: 179984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498477 with bad export cookie 7319248647459376771 [ 3021.174135] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3021.333887] Lustre: server umount lustre-MDT0001 complete [ 3026.309290] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3027.934181] Key type lgssc unregistered [ 3028.124582] LNet: 183018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3028.129659] LNetError: 183018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3028.140835] LNet: Removed LNI 192.168.206.108@tcp [ 3028.664146] Key type .llcrypt unregistered [ 3028.666249] Key type ._llcrypt unregistered [ 3039.781915] Key type ._llcrypt registered [ 3039.783099] Key type .llcrypt registered [ 3039.835935] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3040.198899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3040.249076] alg: No test for adler32 (adler32-zlib) [ 3041.148859] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 3041.275373] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3042.895171] Key type lgssc registered [ 3043.511726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3047.546178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3047.557577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.797436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.700698] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3054.791101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.997854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3056.895228] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3060.269784] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3062.901804] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3066.168818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.370135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.121580] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3072.602827] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3073.584143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3099.105720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3099.112621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.124667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.132727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.133576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.141501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.230437] Lustre: server umount lustre-OST0000 complete [ 3105.251728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.252940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.259196] Lustre: Skipped 1 previous similar message [ 3105.268489] Lustre: Skipped 2 previous similar messages [ 3109.246410] Lustre: server umount lustre-MDT0000 complete [ 3110.369478] LustreError: 184412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.379395] LustreError: 184412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3111.026155] LustreError: 184394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498567 with bad export cookie 4947067063312285381 [ 3111.030164] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.033486] LustreError: 184394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3111.219003] Lustre: server umount lustre-MDT0001 complete [ 3116.125123] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3117.729751] Key type lgssc unregistered [ 3117.906702] LNet: 187969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3117.911805] LNetError: 187969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3117.920660] LNet: Removed LNI 192.168.206.108@tcp [ 3118.427306] Key type .llcrypt unregistered [ 3118.428978] Key type ._llcrypt unregistered [ 3130.858441] Key type ._llcrypt registered [ 3130.859905] Key type .llcrypt registered [ 3130.925503] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3131.353865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3131.405189] alg: No test for adler32 (adler32-zlib) [ 3132.312788] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 3132.448730] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3134.039129] Key type lgssc registered [ 3134.656710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3138.639980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3138.653952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.847738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.793405] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3146.094525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.292163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3148.319098] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3151.852812] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3154.619271] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3157.984173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3158.173995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.989895] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3164.340510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3164.501528] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3180.001498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3180.006188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.014456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3181.197989] Lustre: lustre-OST0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 3182.055695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3182.060285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3183.469243] Lustre: server umount lustre-OST0000 complete [ 3186.910524] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b1e183c4800.ost_server_uuid 50 [ 3187.658986] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3202.082192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.083847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3202.089654] Lustre: Skipped 1 previous similar message [ 3202.095734] Lustre: Skipped 3 previous similar messages [ 3204.143089] Lustre: server umount lustre-MDT0000 complete [ 3205.873234] LustreError: 189343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498661 with bad export cookie 11054077167542510545 [ 3205.880539] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.883364] LustreError: 189343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3206.034634] Lustre: server umount lustre-MDT0001 complete [ 3209.839340] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:24:25 (1787498665) [ 3212.906196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.135959] LustreError: 193276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.145833] LustreError: 193276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3213.173356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3214.857701] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3218.400773] LustreError: 193277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.876612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.080304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3220.713937] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3224.046425] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3226.647414] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3229.855704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.045649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.602981] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3235.943704] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3238.251338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3239.194146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.301190] Lustre: lustre-OST0001: new disk, initializing [ 3239.303069] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3239.305671] Lustre: 196116:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3240.860273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3240.866933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3240.920861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3241.929679] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3246.773331] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3248.608289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3248.612838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.620795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.151656] Lustre: server umount lustre-OST0000 complete [ 3255.271158] LustreError: 195096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.280787] LustreError: 195096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3256.911469] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3257.036225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3257.472626] LustreError: 196304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.852201] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3258.976169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3262.592540] LustreError: 195098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.602876] LustreError: 195098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3270.627385] LustreError: 196304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.631104] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.632330] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3270.632338] Lustre: Skipped 3 previous similar messages [ 3270.637061] LustreError: 196304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3270.648102] Lustre: Skipped 2 previous similar messages [ 3275.431566] Lustre: server umount lustre-OST0001 complete [ 3280.352336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.357040] LustreError: Skipped 1 previous similar message [ 3280.359750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.367843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.371524] Lustre: Skipped 1 previous similar message [ 3283.389291] Lustre: server umount lustre-MDT0000 complete [ 3285.126971] LustreError: 194012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498741 with bad export cookie 11054077167542511602 [ 3285.132853] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.133920] LustreError: 194012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3285.283374] Lustre: server umount lustre-MDT0001 complete [ 3296.036146] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:25:51 (1787498751) [ 3299.130438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.260168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3299.276183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3299.378751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.382993] Lustre: Skipped 1 previous similar message [ 3301.244524] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3305.216750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.269650] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3305.290093] Lustre: 199531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3307.128439] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3310.443918] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3312.995522] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3316.158704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.263553] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3319.043961] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3322.356676] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3323.445100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3325.539067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.576206] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3328.297700] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3331.353172] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3333.115440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.124066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.124220] Lustre: Skipped 2 previous similar messages [ 3333.128041] Lustre: Skipped 1 previous similar message [ 3338.759609] Lustre: server umount lustre-OST0000 complete [ 3341.952072] LustreError: 201382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.574447] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3356.069527] Lustre: server umount lustre-OST0001 complete [ 3364.021490] Lustre: server umount lustre-MDT0000 complete [ 3365.774380] LustreError: 200278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498821 with bad export cookie 11054077167542513002 [ 3365.779851] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.781164] LustreError: 200278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3365.931474] Lustre: server umount lustre-MDT0001 complete [ 3377.030282] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:27:12 (1787498832) [ 3384.534796] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 3389.105646] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3405.446825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.573135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3405.589208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.640324] Lustre: lustre-MDT0000: new disk, initializing [ 3405.662348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.664614] Lustre: Skipped 3 previous similar messages [ 3405.671179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3407.481302] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3412.848878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.921576] Lustre: 207592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3412.961111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3412.965329] Lustre: Skipped 1 previous similar message [ 3413.012600] Lustre: lustre-MDT0001: new disk, initializing [ 3413.070281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3413.075207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3414.824525] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3419.627191] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3422.166782] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3425.350758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3425.504602] Lustre: lustre-OST0000: new disk, initializing [ 3425.508073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3425.512950] Lustre: 209562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3428.214583] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3432.455627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3432.461968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3432.517043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3433.069510] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3434.987870] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3435.099695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3436.869759] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3436.969813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3442.659213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3442.661371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.663794] LustreError: Skipped 2 previous similar messages [ 3442.670958] Lustre: Skipped 4 previous similar messages [ 3442.674719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3442.679790] Lustre: Skipped 11 previous similar messages [ 3452.383151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.463029] Lustre: server umount lustre-OST0000 complete [ 3460.401009] Lustre: server umount lustre-MDT0000 complete [ 3462.126765] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787498918 with bad export cookie 11054077167542514521 [ 3462.132193] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.133385] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3462.285914] Lustre: server umount lustre-MDT0001 complete [ 3466.208577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.417631] LustreError: 211777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.423430] LustreError: 211777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3468.222307] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3472.202643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.375609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.379446] Lustre: Skipped 3 previous similar messages [ 3474.179403] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3477.497122] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3480.052214] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3483.226139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3486.012814] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3489.399449] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3491.324532] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3492.475434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3494.257357] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3498.489224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3502.049219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3505.507713] Lustre: server umount lustre-OST0000 complete [ 3508.238994] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3510.114943] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3534.719039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3537.409256] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3540.868486] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3541.632674] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3543.970831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.974552] Lustre: Skipped 7 previous similar messages [ 3549.031188] Lustre: server umount lustre-OST0000 complete [ 3554.275653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.276242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3554.282781] Lustre: Skipped 5 previous similar messages [ 3554.288171] LustreError: Skipped 1 previous similar message [ 3557.037812] Lustre: server umount lustre-MDT0000 complete [ 3558.839454] LustreError: 215920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499014 with bad export cookie 11054077167542515382 [ 3558.844342] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.847273] LustreError: 215920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3558.998624] Lustre: server umount lustre-MDT0001 complete [ 3563.063645] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:30:18 (1787499018) [ 3566.220073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.474787] LustreError: 217773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.481972] LustreError: 217773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3568.532056] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3572.531172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.546988] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3578.023355] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3580.628301] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3583.892935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3586.921539] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3590.371372] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3592.484932] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3592.614438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3594.588170] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3597.817496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3601.046266] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3601.161124] Lustre: lustre-OST0001: new disk, initializing [ 3601.163361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3601.166838] Lustre: 221162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3601.215900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3601.218163] Lustre: Skipped 5 previous similar messages [ 3604.004852] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3608.603620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3608.607898] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3608.663733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3608.955558] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3611.009774] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3611.140984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3613.066142] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3613.200388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3618.793747] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3618.798390] LustreError: Skipped 1 previous similar message [ 3620.239271] Lustre: server umount lustre-OST0001 complete [ 3623.184960] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3623.313818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3625.328784] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3625.461336] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3650.230365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.182335] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3655.664093] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3655.669502] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3655.669619] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3656.651781] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3657.398798] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3664.397304] Lustre: server umount lustre-OST0001 complete [ 3678.689358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3678.693132] Lustre: Skipped 7 previous similar messages [ 3683.179690] Lustre: server umount lustre-OST0000 complete [ 3685.344533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.352083] Lustre: Skipped 6 previous similar messages [ 3691.326132] Lustre: server umount lustre-MDT0000 complete [ 3693.026763] LustreError: 223522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499149 with bad export cookie 11054077167542516341 [ 3693.033860] LustreError: 223522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3693.033885] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.359109] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3707.472917] Lustre: server umount lustre-MDT0001 complete [ 3718.375866] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:32:53 (1787499173) [ 3721.528165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.680417] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3721.700526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3723.809378] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3727.934732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.987779] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3728.001624] Lustre: 226399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3730.146373] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3733.598203] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3736.224358] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3739.495164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.606510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3742.356358] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3745.802182] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3750.727402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3750.769801] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3753.642312] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3757.193584] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3757.939604] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3759.959352] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3760.092984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3762.071252] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3762.210180] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3764.848592] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b1e07c8f800.ost_server_uuid 50 [ 3765.599523] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b1e07c8f800.ost_server_uuid in FULL state after 0 sec [ 3766.096871] Lustre: Permanently deactivating lustre-OST0001 [ 3766.107144] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3767.183450] Lustre: Permanently reactivating lustre-OST0001 [ 3769.832262] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3769.837257] LustreError: Skipped 6 previous similar messages [ 3774.607305] Lustre: server umount lustre-OST0001 complete [ 3774.955171] LustreError: 228245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3774.965346] LustreError: 228245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 3782.636683] Lustre: server umount lustre-OST0000 complete [ 3790.660870] Lustre: server umount lustre-MDT0000 complete [ 3792.478461] LustreError: 229271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499248 with bad export cookie 11054077167542517615 [ 3792.485399] LustreError: 229271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3792.487107] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.688205] Lustre: server umount lustre-MDT0001 complete [ 3803.998031] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:34:19 (1787499259) [ 3807.659765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.805255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3807.824318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3807.830842] Lustre: Skipped 5 previous similar messages [ 3809.782231] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3813.941758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.003770] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3814.023391] Lustre: 233084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3816.107536] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3819.625797] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3822.262292] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3825.586496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.728388] Lustre: Permanently deactivating lustre-OST0000 [ 3825.738282] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3828.552962] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3831.995813] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3834.369344] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3835.341938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.321832] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3841.856424] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3844.191223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3847.929555] Lustre: Permanently reactivating lustre-OST0000 [ 3847.939364] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3847.945650] Lustre: Skipped 2 previous similar messages [ 3852.823995] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3852.883520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3859.337889] Lustre: server umount lustre-OST0001 complete [ 3867.308725] Lustre: server umount lustre-OST0000 complete [ 3875.458841] Lustre: server umount lustre-MDT0000 complete [ 3877.300398] LustreError: 234953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499333 with bad export cookie 11054077167542519246 [ 3877.307633] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3877.311148] LustreError: 234953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3877.483785] Lustre: server umount lustre-MDT0001 complete [ 3882.560312] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 3884.179983] Key type lgssc unregistered [ 3884.370631] LNet: 237940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3884.376467] LNetError: 237940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3884.385697] LNet: Removed LNI 192.168.206.108@tcp [ 3884.904290] Key type .llcrypt unregistered [ 3884.906543] Key type ._llcrypt unregistered [ 3895.490528] Key type ._llcrypt registered [ 3895.492269] Key type .llcrypt registered [ 3895.559620] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 3896.121303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.146389] alg: No test for adler32 (adler32-zlib) [ 3897.052004] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 3897.193337] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 3898.799200] Key type lgssc registered [ 3899.440497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3904.466248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3904.478821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.702364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3907.668154] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3911.845502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.906297] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3911.922236] Lustre: 239370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3911.932414] Lustre: Permanently deactivating lustre-MDT0001 [ 3911.940781] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3912.110584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3913.339814] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3913.344514] LustreError: 239414:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3913.350487] LustreError: 239414:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3913.355822] LustreError: 239414:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3914.072164] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3917.562625] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3920.165117] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3923.437402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3923.623334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.416062] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3929.836223] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3930.792469] LustreError: 241225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.799220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3930.802845] LustreError: 241225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3933.046128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.153237] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.973596] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 3937.275511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3937.283456] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3939.546865] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3941.399397] Lustre: Permanently reactivating lustre-MDT0001 [ 3941.409400] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3941.413472] Lustre: Skipped 2 previous similar messages [ 3946.635227] Lustre: lustre-MDT0001: Client ea7c1316-0ee7-423b-b0cc-ce599c6708ad (at 192.168.206.8@tcp) reconnecting [ 3949.201354] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3949.944753] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3952.397786] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3952.518029] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3954.538311] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3954.671826] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3955.925322] Lustre: Permanently deactivating lustre-MDT0001 [ 3955.935972] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3955.940121] Lustre: Skipped 2 previous similar messages [ 3957.804549] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3966.948488] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3966.953298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.961645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3967.353764] Lustre: server umount lustre-MDT0000 complete [ 3967.975052] LustreError: 239382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.976418] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.985598] LustreError: 239382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3967.993041] Lustre: Skipped 2 previous similar messages [ 3969.175116] LustreError: 240123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499425 with bad export cookie 10113093485593207705 [ 3969.181747] LustreError: 240123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3969.184411] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3969.381772] Lustre: server umount lustre-MDT0001 complete [ 3977.286786] Lustre: server umount lustre-OST0000 complete [ 3982.513827] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:37:17 (1787499437) [ 3987.575444] Lustre: server umount lustre-OST0001 complete [ 3990.665381] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 3995.235261] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4012.245634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.399597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4012.406302] Lustre: Skipped 2 previous similar messages [ 4012.420751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4012.482166] Lustre: lustre-MDT0000: new disk, initializing [ 4012.522242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.532830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4014.422298] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4020.001837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.077332] Lustre: 247629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4020.122352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4020.126031] Lustre: Skipped 1 previous similar message [ 4020.177040] Lustre: lustre-MDT0001: new disk, initializing [ 4020.241296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4020.248225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4022.153200] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4027.016197] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4029.614371] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4032.916698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4033.081843] Lustre: lustre-OST0000: new disk, initializing [ 4033.085356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4033.090157] Lustre: 249596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4033.145353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.149285] Lustre: Skipped 1 previous similar message [ 4035.787137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4035.790807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4035.837708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4035.893152] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4040.727268] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.689536] LustreError: 247636:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4046.923486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4047.037805] Lustre: lustre-OST0001: new disk, initializing [ 4047.041548] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4047.047276] Lustre: 250711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4049.903265] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4051.783120] LustreError: 247636:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4051.795460] LustreError: 247637:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4053.521341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4053.524493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4053.575096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4054.759157] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4055.527144] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4061.887121] LustreError: 247637:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4063.725335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4063.726858] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.730765] LustreError: Skipped 1 previous similar message [ 4063.740375] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4063.745982] Lustre: Skipped 1 previous similar message [ 4068.848261] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4068.852542] Lustre: Skipped 2 previous similar messages [ 4068.942762] Lustre: server umount lustre-OST0001 complete [ 4069.506780] LustreError: 249580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.958478] LustreError: 249935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.967915] LustreError: 249935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4079.088850] LustreError: 249580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.098725] LustreError: 249580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4082.145767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.145967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.150423] LustreError: Skipped 1 previous similar message [ 4082.151429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.159030] Lustre: Skipped 2 previous similar messages [ 4082.167868] Lustre: Skipped 1 previous similar message [ 4087.795231] Lustre: server umount lustre-OST0000 complete [ 4091.872629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.877665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.886571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.890525] Lustre: Skipped 2 previous similar messages [ 4095.873458] Lustre: server umount lustre-MDT0000 complete [ 4097.715577] LustreError: 248425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499553 with bad export cookie 10113093485593210260 [ 4097.725421] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.351186] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4112.492132] Lustre: server umount lustre-MDT0001 complete [ 4117.549078] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4119.141311] Key type lgssc unregistered [ 4119.324669] LNet: 252746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4119.331715] LNetError: 252746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4119.342918] LNet: Removed LNI 192.168.206.108@tcp [ 4119.863463] Key type .llcrypt unregistered [ 4119.865496] Key type ._llcrypt unregistered [ 4139.913718] Key type ._llcrypt registered [ 4139.915225] Key type .llcrypt registered [ 4139.971444] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4140.400348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4140.437441] alg: No test for adler32 (adler32-zlib) [ 4141.355133] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4141.487987] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4143.095159] Key type lgssc registered [ 4143.746406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.786456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4147.798066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.943422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4148.957807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4149.043147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.033875] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4155.263949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.328231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4155.348564] Lustre: 254937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4155.526700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.489327] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4161.089996] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4163.822151] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4167.164668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.283169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4167.414440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.249853] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4172.364961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4173.878288] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4218.339089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4218.340673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.343679] LustreError: Skipped 1 previous similar message [ 4218.344640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4218.352268] Lustre: Skipped 1 previous similar message [ 4219.449186] Lustre: server umount lustre-OST0000 complete [ 4221.535962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4227.335142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.524874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.132424] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4233.527922] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4238.825051] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.832427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.832717] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4238.844880] Lustre: Skipped 1 previous similar message [ 4269.540708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.540802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.545294] LustreError: Skipped 1 previous similar message [ 4269.553692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.558929] Lustre: Skipped 2 previous similar messages [ 4273.839471] Lustre: server umount lustre-OST0000 complete [ 4278.752499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4278.757344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.764695] Lustre: Skipped 1 previous similar message [ 4278.767992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4281.777189] Lustre: server umount lustre-MDT0000 complete [ 4283.513940] LustreError: 254932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499739 with bad export cookie 3319577304130118178 [ 4283.520557] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4283.520930] LustreError: 254932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4283.660649] Lustre: server umount lustre-MDT0001 complete [ 4288.522959] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4290.031368] Key type lgssc unregistered [ 4290.190641] LNet: 260358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4290.197567] LNetError: 260358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4290.210585] LNet: Removed LNI 192.168.206.108@tcp [ 4290.672679] Key type .llcrypt unregistered [ 4290.674696] Key type ._llcrypt unregistered [ 4301.911176] Key type ._llcrypt registered [ 4301.912759] Key type .llcrypt registered [ 4301.961683] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4302.354286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4302.363450] alg: No test for adler32 (adler32-zlib) [ 4303.248509] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4303.364030] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4304.967145] Key type lgssc registered [ 4305.547083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4309.210590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4309.222340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.407566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.075255] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4316.032448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.189663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4317.948151] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4321.292560] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4323.846318] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4326.995275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.185475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.752313] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4333.140599] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4335.403764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4356.065297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4356.070732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.080080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4357.090096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4357.090766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4357.094741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.269387] Lustre: server umount lustre-OST0000 complete [ 4362.210569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4362.210830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.212282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.212290] Lustre: Skipped 1 previous similar message [ 4362.229950] Lustre: Skipped 1 previous similar message [ 4367.212055] Lustre: server umount lustre-MDT0000 complete [ 4367.328773] LustreError: 261750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.338664] LustreError: 261750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4368.937138] LustreError: 263593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499824 with bad export cookie 11726539045450524144 [ 4368.943740] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.944086] LustreError: 263593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4369.081495] Lustre: server umount lustre-MDT0001 complete [ 4373.707243] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4375.237466] Key type lgssc unregistered [ 4375.397687] LNet: 266079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4375.402334] LNetError: 266079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4375.414588] LNet: Removed LNI 192.168.206.108@tcp [ 4375.888990] Key type .llcrypt unregistered [ 4375.891303] Key type ._llcrypt unregistered [ 4385.044178] Key type ._llcrypt registered [ 4385.045301] Key type .llcrypt registered [ 4385.097820] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4385.488760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4385.527165] alg: No test for adler32 (adler32-zlib) [ 4386.426950] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4386.551425] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4388.159169] Key type lgssc registered [ 4388.762512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4392.645841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4392.657348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.855906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.869997] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4400.038536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.234767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4402.274446] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4405.825408] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4408.529208] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.758246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.942073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.535216] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4418.022958] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4418.092904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4433.376312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4433.380749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.388843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4435.426223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4435.426953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4435.434394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.560121] Lustre: server umount lustre-OST0000 complete [ 4440.545351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.546534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.552692] Lustre: Skipped 1 previous similar message [ 4440.558938] Lustre: Skipped 2 previous similar messages [ 4445.664920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.668373] Lustre: Skipped 1 previous similar message [ 4446.508165] Lustre: server umount lustre-MDT0000 complete [ 4448.271081] LustreError: 267259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499904 with bad export cookie 16991411793954637198 [ 4448.276204] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4448.278030] LustreError: 267259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4448.426902] Lustre: server umount lustre-MDT0001 complete [ 4453.442343] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4454.956586] Key type lgssc unregistered [ 4455.126699] LNet: 270977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4455.131491] LNetError: 270977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4455.141651] LNet: Removed LNI 192.168.206.108@tcp [ 4455.579056] Key type .llcrypt unregistered [ 4455.581248] Key type ._llcrypt unregistered [ 4467.235591] Key type ._llcrypt registered [ 4467.236738] Key type .llcrypt registered [ 4467.292333] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4467.706129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4467.716605] alg: No test for adler32 (adler32-zlib) [ 4468.617311] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4468.746534] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4470.351181] Key type lgssc registered [ 4470.973208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4474.868413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4474.879348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.063303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4478.077794] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4482.221180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.379372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4484.276857] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4487.854618] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4490.594198] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4493.863424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4494.046330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.797514] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4500.331691] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4501.233381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4521.952496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4521.957085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.965287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4522.979962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4522.980269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4522.984599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.636578] Lustre: server umount lustre-OST0000 complete [ 4528.608450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4528.613068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.621287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4528.625308] Lustre: Skipped 1 previous similar message [ 4533.217432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4533.219145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.648456] Lustre: server umount lustre-MDT0000 complete [ 4536.376290] LustreError: 272398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787499992 with bad export cookie 9806939184657239625 [ 4536.382145] LustreError: 272398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4536.383163] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4536.541756] Lustre: server umount lustre-MDT0001 complete [ 4541.401225] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4543.025266] Key type lgssc unregistered [ 4543.225830] LNet: 276698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4543.230387] LNetError: 276698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4543.240657] LNet: Removed LNI 192.168.206.108@tcp [ 4543.738556] Key type .llcrypt unregistered [ 4543.740675] Key type ._llcrypt unregistered [ 4552.920136] Key type ._llcrypt registered [ 4552.921326] Key type .llcrypt registered [ 4552.976967] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4553.479732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4553.488771] alg: No test for adler32 (adler32-zlib) [ 4554.379972] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4554.504608] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4556.111172] Key type lgssc registered [ 4556.734884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4560.532606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4560.543973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.755449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4563.668472] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4567.809380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.006212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4569.918384] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4573.522728] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4576.209263] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4579.513133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4579.695772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.489528] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4586.056626] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4586.864113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4602.336275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4602.340901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.348914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.364419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.365367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4603.372671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.144650] Lustre: server umount lustre-OST0000 complete [ 4608.992394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4608.997153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.005182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4609.009480] Lustre: Skipped 1 previous similar message [ 4613.600993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.602104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.721691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.725119] Lustre: Skipped 2 previous similar messages [ 4623.327154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4623.411840] Lustre: server umount lustre-MDT0000 complete [ 4623.842199] LustreError: 277919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.852241] LustreError: 277919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4625.162096] LustreError: 277879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500081 with bad export cookie 6383543145424761594 [ 4625.168103] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.168827] LustreError: 277879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4625.325805] Lustre: server umount lustre-MDT0001 complete [ 4630.288741] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4631.704608] Key type lgssc unregistered [ 4631.873581] LNet: 281632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4631.878421] LNetError: 281632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4631.889554] LNet: Removed LNI 192.168.206.108@tcp [ 4632.335149] Key type .llcrypt unregistered [ 4632.337377] Key type ._llcrypt unregistered [ 4637.168663] Key type ._llcrypt registered [ 4637.170279] Key type .llcrypt registered [ 4637.225552] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4644.886534] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4652.729323] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4653.039329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4653.055289] alg: No test for adler32 (adler32-zlib) [ 4653.919869] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4654.032276] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4655.615174] Key type lgssc registered [ 4656.182176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4672.908291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4672.917802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4674.033938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4674.049578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4674.092849] Lustre: lustre-MDT0000: new disk, initializing [ 4674.116786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.128179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4675.965812] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4681.518067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.591303] Lustre: 285169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4681.635703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4681.639274] Lustre: Skipped 1 previous similar message [ 4681.688208] Lustre: lustre-MDT0001: new disk, initializing [ 4681.740115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4681.764172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4681.771840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4683.651583] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4688.463270] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4691.031078] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4694.298763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.478263] Lustre: lustre-OST0000: new disk, initializing [ 4694.480994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4694.484897] Lustre: 287141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4694.536038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.298612] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4697.676976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4697.683149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4697.743579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4702.217934] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4704.224404] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4704.353478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4706.369487] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4706.511812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4708.325887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4708.327107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4708.330645] LustreError: Skipped 1 previous similar message [ 4708.330933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4708.338044] Lustre: Skipped 1 previous similar message [ 4713.447360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4713.451068] Lustre: Skipped 2 previous similar messages [ 4713.581824] Lustre: server umount lustre-OST0000 complete [ 4717.536526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4717.541971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.551389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4718.564754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.601651] Lustre: server umount lustre-MDT0000 complete [ 4723.403299] LustreError: 285161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500179 with bad export cookie 17190662060254908713 [ 4723.411650] LustreError: 285161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4723.413141] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4723.564665] Lustre: server umount lustre-MDT0001 complete [ 4727.552743] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:49:42 (1787500182) [ 4730.753765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.048475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.992695] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4736.481518] LustreError: 289549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4736.490986] LustreError: 289549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4737.092120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.283435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4739.258468] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4742.730041] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4745.354908] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4748.591501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4748.789871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.465882] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4754.912515] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4772.326768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.329916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.331647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4772.331662] Lustre: Skipped 2 previous similar messages [ 4772.334040] Lustre: Skipped 1 previous similar message [ 4772.347858] LustreError: Skipped 1 previous similar message [ 4774.508295] Lustre: server umount lustre-OST0000 complete [ 4777.443344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.452466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.456097] Lustre: Skipped 2 previous similar messages [ 4782.561686] LustreError: 289549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.568273] LustreError: 289549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4782.582166] Lustre: server umount lustre-MDT0000 complete [ 4784.351959] LustreError: 291390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500240 with bad export cookie 17190662060254909595 [ 4784.360154] LustreError: 291390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4784.361935] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4784.532271] Lustre: server umount lustre-MDT0001 complete [ 4789.378917] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 4790.955454] Key type lgssc unregistered [ 4791.137562] LNet: 293108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4791.142947] LNetError: 293108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4791.152737] LNet: Removed LNI 192.168.206.108@tcp [ 4791.634938] Key type .llcrypt unregistered [ 4791.637024] Key type ._llcrypt unregistered [ 4803.923158] Key type ._llcrypt registered [ 4803.924474] Key type .llcrypt registered [ 4803.981609] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4812.093589] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4812.409723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4812.453248] alg: No test for adler32 (adler32-zlib) [ 4813.328806] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 4813.441832] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 4815.039147] Key type lgssc registered [ 4815.666458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4837.815681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4837.827477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.971685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4838.986132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4839.044549] Lustre: lustre-MDT0000: new disk, initializing [ 4839.070595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.078775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4841.047193] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4846.672366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.738940] Lustre: 296904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4846.784020] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4846.787687] Lustre: Skipped 1 previous similar message [ 4846.838495] Lustre: lustre-MDT0001: new disk, initializing [ 4846.891632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4846.914940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4846.921944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4848.752824] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4853.727584] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4856.442750] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4859.746399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.923739] Lustre: lustre-OST2710: new disk, initializing [ 4859.926951] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4859.931769] Lustre: 298871:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4859.988432] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4862.657347] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4867.691623] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4869.142007] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4869.148163] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4869.207364] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4871.031233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4871.148473] Lustre: lustre-OST03e8: new disk, initializing [ 4871.151693] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4871.156539] Lustre: 299937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4871.216820] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4873.947983] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4877.529354] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4878.876993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4878.883846] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4878.951965] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4881.042435] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4881.173663] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4883.157565] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4883.279616] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4885.322037] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4885.450775] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4887.445964] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4887.579815] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4889.064770] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.069518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4889.072049] Lustre: Skipped 3 previous similar messages [ 4889.075654] Lustre: Skipped 2 previous similar messages [ 4894.179448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4894.183082] Lustre: Skipped 2 previous similar messages [ 4895.186463] Lustre: server umount lustre-MDT0000 complete [ 4896.983408] LustreError: 296897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500352 with bad export cookie 13013608848558881878 [ 4896.989870] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4896.990707] LustreError: 296897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4897.160664] Lustre: server umount lustre-MDT0001 complete [ 4908.572428] Lustre: server umount lustre-OST2710 complete [ 4920.704607] Lustre: server umount lustre-OST03e8 complete [ 4929.267466] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 4933.882891] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 4951.303584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.455458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4951.472592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4951.527155] Lustre: lustre-MDT0000: new disk, initializing [ 4951.559370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.571144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4953.542074] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4958.931915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.988535] Lustre: 305174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.155815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4961.196619] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4966.232221] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4968.853984] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4972.107330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4972.279215] Lustre: lustre-OST0000: new disk, initializing [ 4972.281984] Lustre: Skipped 1 previous similar message [ 4972.285072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4972.289331] Lustre: Skipped 2 previous similar messages [ 4972.293342] Lustre: 307143:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4972.346600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.350216] Lustre: Skipped 1 previous similar message [ 4974.099922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4974.105631] Lustre: Skipped 1 previous similar message [ 4974.109597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4974.162746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4975.106105] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 4979.908973] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4982.004633] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4982.141888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4984.156452] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4984.292721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4989.414556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4989.416473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.419121] LustreError: Skipped 1 previous similar message [ 4989.427174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4989.432484] Lustre: Skipped 3 previous similar messages [ 4991.340472] Lustre: server umount lustre-OST0000 complete [ 4994.531178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.533148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4994.538574] Lustre: Skipped 2 previous similar messages [ 4999.358503] Lustre: server umount lustre-MDT0000 complete [ 5001.130380] LustreError: 305168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500457 with bad export cookie 13013608848558883551 [ 5001.137152] LustreError: 305168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5001.139636] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5001.296556] Lustre: server umount lustre-MDT0001 complete [ 5005.289607] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:54:20 (1787500460) [ 5006.082136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5006.975760] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:54:22 (1787500462) [ 5008.447746] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5021.046485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.164985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5021.181037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5021.259213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.258213] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5027.387752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.443591] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5027.459813] Lustre: 310995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5029.519342] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5046.759518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5046.867846] Lustre: Denying initial registration attempt from nid 192.168.206.108@tcp, specified as failover [ 5046.875334] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5046.880256] LustreError: 312448:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 5046.884264] LustreError: 312448:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5046.887661] LustreError: 312448:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5046.930577] Lustre: server umount lustre-OST0000 complete [ 5046.932505] LustreError: 312448:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5051.875541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.876578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.881553] Lustre: Skipped 1 previous similar message [ 5051.887291] Lustre: Skipped 2 previous similar messages [ 5056.311291] Lustre: server umount lustre-MDT0000 complete [ 5056.994334] LustreError: 311007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.008752] LustreError: 311007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5058.131110] LustreError: 311738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500514 with bad export cookie 13013608848558884440 [ 5058.139392] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5058.141418] LustreError: 311738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5058.302735] Lustre: server umount lustre-MDT0001 complete [ 5063.511763] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5065.106296] Key type lgssc unregistered [ 5065.292617] LNet: 313571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5065.298281] LNetError: 313571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5065.308693] LNet: Removed LNI 192.168.206.108@tcp [ 5065.812666] Key type .llcrypt unregistered [ 5065.814811] Key type ._llcrypt unregistered [ 5070.867188] Key type ._llcrypt registered [ 5070.868897] Key type .llcrypt registered [ 5070.935923] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5071.298140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5071.312287] alg: No test for adler32 (adler32-zlib) [ 5072.226131] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5072.352199] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5073.951152] Key type lgssc registered [ 5074.589485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5103.225497] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5107.208347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5107.219992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5108.351769] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5108.360802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5108.450566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.512889] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5114.646118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.700823] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5114.720498] Lustre: 316314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5114.882994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5116.826513] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5120.295588] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5122.910113] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5139.236844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5139.347841] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5139.461664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.274067] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5145.734770] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5146.810775] Lustre: server umount lustre-OST0000 complete [ 5150.688538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5150.695524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.707298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.226440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.227085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.825228] Lustre: server umount lustre-MDT0000 complete [ 5156.611774] LustreError: 316349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500612 with bad export cookie 17524311156889521165 [ 5156.620967] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.622224] LustreError: 316349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5156.810135] Lustre: server umount lustre-MDT0001 complete [ 5161.872201] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5163.478420] Key type lgssc unregistered [ 5163.671758] LNet: 319883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5163.678633] LNetError: 319883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5163.691710] LNet: Removed LNI 192.168.206.108@tcp [ 5164.194885] Key type .llcrypt unregistered [ 5164.198113] Key type ._llcrypt unregistered [ 5176.491267] Key type ._llcrypt registered [ 5176.493871] Key type .llcrypt registered [ 5176.564343] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5176.997952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5177.036401] alg: No test for adler32 (adler32-zlib) [ 5177.927350] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5178.033218] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5179.623150] Key type lgssc registered [ 5180.242447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5184.192330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5184.203546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5185.417983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.350872] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5191.483306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.684781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5193.647975] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5197.278964] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5199.879329] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5205.220259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.420939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.249268] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5211.773035] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5221.862313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.866164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5221.871419] Lustre: Skipped 2 previous similar messages [ 5221.875460] Lustre: Skipped 1 previous similar message [ 5223.565854] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 5226.980136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5226.983833] Lustre: Skipped 2 previous similar messages [ 5227.324787] Lustre: server umount lustre-MDT0000 complete [ 5228.671761] LustreError: 321270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.682876] LustreError: 321270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5229.153825] LustreError: 321257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500685 with bad export cookie 11138017519765858240 [ 5229.161200] LustreError: 321257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5229.163615] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.096655] LustreError: 322472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5232.097274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.118853] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5233.791885] LustreError: 322472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.421003] Lustre: server umount lustre-MDT0001 complete [ 5238.575695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5242.736167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.624182] LustreError: 320587:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009723ee21 x1874330264822784/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5254.805570] LustreError: 323106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5254.815941] LustreError: 323106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5254.851688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.898398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5256.739231] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5260.003884] LustreError: 323107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.014024] LustreError: 323107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5260.945290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.121268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5263.084985] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5264.238402] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5266.764228] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5267.548736] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5270.223561] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5275.213819] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 5279.718148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.719911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.725588] Lustre: Skipped 1 previous similar message [ 5279.731607] Lustre: Skipped 3 previous similar messages [ 5283.181127] Lustre: server umount lustre-OST0000 complete [ 5286.880381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5286.885774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.952989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.956929] Lustre: Skipped 2 previous similar messages [ 5291.185227] Lustre: server umount lustre-MDT0000 complete [ 5292.937046] LustreError: 321256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500748 with bad export cookie 11138017519765865394 [ 5292.942384] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5292.945515] LustreError: 321256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5293.073695] Lustre: server umount lustre-MDT0001 complete [ 5298.238406] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5299.637118] Key type lgssc unregistered [ 5299.793637] LNet: 327544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5299.798233] LNetError: 327544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5299.809549] LNet: Removed LNI 192.168.206.108@tcp [ 5300.244042] Key type .llcrypt unregistered [ 5300.246704] Key type ._llcrypt unregistered [ 5312.572311] Key type ._llcrypt registered [ 5312.573973] Key type .llcrypt registered [ 5312.634281] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5312.989045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5313.040961] alg: No test for adler32 (adler32-zlib) [ 5313.907537] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5313.998684] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5315.583158] Key type lgssc registered [ 5316.154995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5319.848673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5319.862133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.064676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.929867] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5326.991482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.196895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5329.019683] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5332.486929] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.162183] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5337.568506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5337.573895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.583206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.593348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.594951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.825089] Lustre: server umount lustre-MDT0000 complete [ 5343.712830] LustreError: 328938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.726856] LustreError: 328938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5344.550966] LustreError: 329671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500800 with bad export cookie 4571845471127413195 [ 5344.555897] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.559359] LustreError: 329671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5344.712482] Lustre: server umount lustre-MDT0001 complete [ 5351.579299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.703274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5351.720490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5351.832792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.659744] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5357.742908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.804620] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5357.823111] Lustre: 331524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5357.979395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5359.679066] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5363.211498] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5365.918631] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5371.620740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5371.736907] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5371.745990] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5371.751965] LustreError: 333409:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5371.756890] LustreError: 333409:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5371.761170] LustreError: 333409:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5371.802592] Lustre: server umount lustre-OST0000 complete [ 5371.805370] LustreError: 333409:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5377.764295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5377.878549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5377.988869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.579725] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5383.894299] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5385.269091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5387.458572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5387.514413] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5387.571069] Lustre: lustre-OST0001: new disk, initializing [ 5387.573957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5387.578467] Lustre: 334950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5387.630394] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5390.226434] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5392.419869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5392.430579] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5392.474787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5393.521720] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5397.477121] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5397.480370] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.481966] LustreError: Skipped 1 previous similar message [ 5397.490637] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5397.496483] Lustre: Skipped 2 previous similar messages [ 5400.972898] Lustre: server umount lustre-OST0001 complete [ 5402.598958] LustreError: 333875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.613604] LustreError: 333875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5405.666460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5405.666556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.671064] LustreError: Skipped 1 previous similar message [ 5405.682247] Lustre: Skipped 1 previous similar message [ 5405.683322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.691804] Lustre: Skipped 1 previous similar message [ 5407.719288] LustreError: 333875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.730335] LustreError: 333875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5408.869778] Lustre: server umount lustre-OST0000 complete [ 5412.834555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.836067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.843234] Lustre: Skipped 2 previous similar messages [ 5412.847088] Lustre: Skipped 2 previous similar messages [ 5416.826864] Lustre: server umount lustre-MDT0000 complete [ 5417.953889] LustreError: 332280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.965320] LustreError: 332280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5418.566891] LustreError: 331562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787500874 with bad export cookie 4571845471127413748 [ 5418.573912] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.575409] LustreError: 331562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5418.729659] Lustre: server umount lustre-MDT0001 complete [ 5423.360524] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5424.857506] Key type lgssc unregistered [ 5425.030594] LNet: 336919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5425.036177] LNetError: 336919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5425.045519] LNet: Removed LNI 192.168.206.108@tcp [ 5425.532669] Key type .llcrypt unregistered [ 5425.534734] Key type ._llcrypt unregistered [ 5463.662498] Key type ._llcrypt registered [ 5463.663982] Key type .llcrypt registered [ 5463.717578] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5471.828094] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5472.238378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5472.252570] alg: No test for adler32 (adler32-zlib) [ 5473.134087] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5473.225742] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5474.807147] Key type lgssc registered [ 5475.416808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5492.946169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5492.956632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5494.088711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5494.109250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5494.172225] Lustre: lustre-MDT0000: new disk, initializing [ 5494.206705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.219911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5496.202577] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5501.785052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.861605] Lustre: 341820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5501.937336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5501.944701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5501.948013] Lustre: Skipped 1 previous similar message [ 5501.994379] Lustre: lustre-MDT0001: new disk, initializing [ 5502.053100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5502.078288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5502.084863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5504.063577] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5509.097339] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5511.870794] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5515.224641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5515.402855] Lustre: lustre-OST0000: new disk, initializing [ 5515.406210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5515.411135] Lustre: 343790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5515.464297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.315493] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5521.431201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5521.438235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5521.486699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5523.332822] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5525.368781] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5525.509666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5527.538966] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5527.669676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5531.619543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5531.624271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.628242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5531.631396] Lustre: Skipped 1 previous similar message [ 5534.767458] Lustre: server umount lustre-OST0000 complete [ 5536.738392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.739363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5536.745664] Lustre: Skipped 1 previous similar message [ 5536.752177] Lustre: Skipped 2 previous similar messages [ 5541.857804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5541.861683] Lustre: Skipped 1 previous similar message [ 5542.782301] Lustre: server umount lustre-MDT0000 complete [ 5544.607571] LustreError: 341813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501000 with bad export cookie 2565042807979481021 [ 5544.614632] LustreError: 341813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5544.615870] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.766573] Lustre: server umount lustre-MDT0001 complete [ 5548.985660] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:03:24 (1787501004) [ 5552.681725] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:03:28 (1787501008) [ 5556.862678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.136335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.013921] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5562.336891] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.343737] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5562.891694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.087696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5565.087360] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5568.741293] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5571.513474] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5574.834373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.027537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.760219] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5581.440544] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5595.807401] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5596.694966] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5597.532408] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5598.411382] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5599.368977] Lustre: Failing over lustre-MDT0000 [ 5599.600974] Lustre: server umount lustre-MDT0000 complete [ 5603.469207] LustreError: 346539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5603.812332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.819935] Lustre: Skipped 2 previous similar messages [ 5608.574960] LustreError: 347192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.585420] LustreError: 347192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5613.696531] LustreError: 347192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.708057] LustreError: 347192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5613.887601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.952741] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.095160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.149029] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5616.038407] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5618.816219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5619.193183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5619.208491] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5619.238513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5621.690919] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5622.478265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5624.752515] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5625.727894] Lustre: Failing over lustre-MDT0000 [ 5626.033611] Lustre: server umount lustre-MDT0000 complete [ 5629.068706] LustreError: 349030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.415173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.417603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5629.426069] Lustre: Skipped 2 previous similar messages [ 5629.431071] LustreError: Skipped 1 previous similar message [ 5639.295641] LustreError: 349030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.306175] LustreError: 349030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5640.332496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.402817] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.588174] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5642.506041] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5644.416665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5645.805199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5645.809029] Lustre: Skipped 2 previous similar messages [ 5645.824502] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5645.854768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5648.509906] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5649.315851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5656.036681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5656.037361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.039077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.054190] Lustre: Skipped 2 previous similar messages [ 5658.223301] Lustre: server umount lustre-MDT0000 complete [ 5659.965149] LustreError: 348383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501115 with bad export cookie 2565042807979483716 [ 5659.971648] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5659.975484] LustreError: 348383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5660.139615] Lustre: server umount lustre-MDT0001 complete [ 5671.301272] Lustre: server umount lustre-OST0000 complete [ 5678.680919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.902230] LustreError: 352895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.907777] LustreError: 352895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5678.928281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.930606] Lustre: Skipped 1 previous similar message [ 5680.791683] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5684.862213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.956208] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5690.417170] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5693.122324] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5696.375771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5699.160210] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5702.634236] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5703.722975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5714.916141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.916236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5714.925744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5714.935882] Lustre: Skipped 3 previous similar messages [ 5716.643526] Lustre: server umount lustre-OST0000 complete [ 5720.555359] Lustre: server umount lustre-MDT0000 complete [ 5722.316896] LustreError: 352875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501178 with bad export cookie 2565042807979484437 [ 5722.322142] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.323817] LustreError: 352875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5722.478718] Lustre: server umount lustre-MDT0001 complete [ 5727.588582] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5729.018464] Key type lgssc unregistered [ 5729.187649] LNet: 356450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5729.192410] LNetError: 356450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5729.204560] LNet: Removed LNI 192.168.206.108@tcp [ 5729.661923] Key type .llcrypt unregistered [ 5729.664087] Key type ._llcrypt unregistered [ 5742.510689] Key type ._llcrypt registered [ 5742.512386] Key type .llcrypt registered [ 5742.579909] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5743.001368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5743.008289] alg: No test for adler32 (adler32-zlib) [ 5743.882042] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5743.977384] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5745.575190] Key type lgssc registered [ 5746.206641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5750.120858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5750.131660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.312160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5753.167769] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5757.272779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.451729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.461271] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5763.046420] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5765.812823] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5769.157789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.367156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.110470] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5775.766647] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5776.562704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5793.254905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5793.255252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.258055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.272152] Lustre: Skipped 1 previous similar message [ 5796.544621] Lustre: server umount lustre-MDT0000 complete [ 5798.338090] LustreError: 357924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501254 with bad export cookie 5150698736683753744 [ 5798.342426] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5798.345690] LustreError: 357924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5798.368444] LustreError: 357967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.369297] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.376385] LustreError: 357967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5798.381469] Lustre: Skipped 1 previous similar message [ 5798.386662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5798.388788] Lustre: Skipped 2 previous similar messages [ 5798.503866] Lustre: server umount lustre-MDT0001 complete [ 5806.516482] Lustre: server umount lustre-OST0000 complete [ 5812.906562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.143359] LustreError: 361671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.183654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.891815] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5818.336799] LustreError: 361672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.901430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.064144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.803742] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5824.178197] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5826.727844] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5829.888026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5830.070583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.642565] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5835.967599] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5838.251763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5849.573588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.575195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5849.581015] Lustre: Skipped 2 previous similar messages [ 5849.588288] Lustre: Skipped 2 previous similar messages [ 5854.689447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5854.693114] Lustre: Skipped 1 previous similar message [ 5854.893751] Lustre: server umount lustre-MDT0000 complete [ 5856.600885] LustreError: 361653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501312 with bad export cookie 5150698736683754843 [ 5856.605941] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5856.608003] LustreError: 361653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5856.746742] Lustre: server umount lustre-MDT0001 complete [ 5868.788248] Lustre: server umount lustre-OST0000 complete [ 5875.927760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5876.146176] LustreError: 365577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.180772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.956306] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5881.313097] LustreError: 365578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5881.987058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.953565] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5887.394065] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5890.121764] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5893.386699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.136883] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 5899.651610] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5901.805482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5906.916162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5906.916447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5906.924389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5908.643287] Lustre: server umount lustre-OST0000 complete [ 5912.559325] Lustre: server umount lustre-MDT0000 complete [ 5913.057536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5913.057879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.069546] LustreError: 365578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.078561] LustreError: 365578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5914.275154] LustreError: 365559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501370 with bad export cookie 5150698736683755851 [ 5914.282065] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.282170] LustreError: 365559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5916.490331] Lustre: server umount lustre-MDT0001 complete [ 5921.574805] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5923.082771] Key type lgssc unregistered [ 5923.252695] LNet: 369112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5923.258203] LNetError: 369112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5923.271540] LNet: Removed LNI 192.168.206.108@tcp [ 5923.740770] Key type .llcrypt unregistered [ 5923.743118] Key type ._llcrypt unregistered [ 5936.321954] Key type ._llcrypt registered [ 5936.323194] Key type .llcrypt registered [ 5936.378422] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5936.851770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5936.889618] alg: No test for adler32 (adler32-zlib) [ 5937.739336] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5937.820727] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5939.399164] Key type lgssc registered [ 5939.971246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5943.662250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5943.672417] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.677795] LustreError: 370531:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5943.689688] LustreError: 370531:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5943.694200] LustreError: 370531:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5943.698816] LustreError: 370531:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5943.703992] LustreError: 370531:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5947.218939] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5947.223918] LustreError: 370927:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5947.235481] LustreError: 370927:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5947.240098] LustreError: 370927:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5947.244896] LustreError: 370927:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5947.250264] LustreError: 370927:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5956.258461] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 5957.812766] Key type lgssc unregistered [ 5957.995631] LNet: 371932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5958.000350] LNetError: 371932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5958.008682] LNet: Removed LNI 192.168.206.108@tcp [ 5958.480476] Key type .llcrypt unregistered [ 5958.482556] Key type ._llcrypt unregistered [ 5966.971106] Key type ._llcrypt registered [ 5966.972836] Key type .llcrypt registered [ 5967.041282] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 5975.220894] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 5975.583389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5975.589241] alg: No test for adler32 (adler32-zlib) [ 5976.478448] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 5976.608283] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 5978.207176] Key type lgssc registered [ 5978.820902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5996.574587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5996.586491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.702122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5997.714206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5997.776061] Lustre: lustre-MDT0000: new disk, initializing [ 5997.812577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5997.824263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5999.855708] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6005.597437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.670858] Lustre: 375062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6005.750488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6005.755081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6005.758877] Lustre: Skipped 1 previous similar message [ 6005.809435] Lustre: lustre-MDT0001: new disk, initializing [ 6005.865284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6005.886940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6005.892502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6007.782473] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6012.837365] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6015.579287] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6018.908994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6019.081139] Lustre: lustre-OST0000: new disk, initializing [ 6019.084581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6019.090223] Lustre: 377030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6019.149895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.065780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6021.070461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6021.122191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6021.906215] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6026.940955] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6029.101712] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6029.236971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6031.255660] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6031.403189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6036.454340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6036.455948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.459146] LustreError: Skipped 1 previous similar message [ 6036.459881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.466560] Lustre: Skipped 1 previous similar message [ 6038.448992] Lustre: server umount lustre-OST0000 complete [ 6041.573105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.573258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6041.574242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6041.574251] Lustre: Skipped 1 previous similar message [ 6041.580736] Lustre: Skipped 1 previous similar message [ 6046.538799] Lustre: server umount lustre-MDT0000 complete [ 6048.378126] LustreError: 375856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501504 with bad export cookie 9013030356661079107 [ 6048.385317] LustreError: 375856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6048.387627] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6048.571106] Lustre: server umount lustre-MDT0001 complete [ 6052.761548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6053.624049] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:11:49 (1787501509) [ 6056.846319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.086855] LustreError: 379489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.095624] LustreError: 379489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6057.125727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.058934] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6062.561790] LustreError: 379508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.208909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.398169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6065.383257] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6068.924478] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6071.469639] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6074.767627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6074.974962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.814263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6081.324189] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.602730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.719958] Lustre: lustre-OST0001: new disk, initializing [ 6084.723303] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6084.728557] Lustre: 382323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6087.592673] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6092.450928] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6092.825234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6092.829179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6092.883554] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6097.898139] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6097.902709] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.906843] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6097.909944] Lustre: Skipped 1 previous similar message [ 6097.913605] Lustre: Skipped 1 previous similar message [ 6100.683607] Lustre: server umount lustre-OST0001 complete [ 6103.024088] LustreError: 383045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.033457] LustreError: 383045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6108.137187] LustreError: 381309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.146565] LustreError: 381309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6113.259404] LustreError: 381310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.269202] LustreError: 381310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6114.786124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.790633] LustreError: Skipped 1 previous similar message [ 6114.793328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.801265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.804841] Lustre: Skipped 1 previous similar message [ 6119.534295] Lustre: server umount lustre-OST0000 complete [ 6123.491460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.493120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.499458] Lustre: Skipped 2 previous similar messages [ 6123.503280] Lustre: Skipped 2 previous similar messages [ 6127.671880] Lustre: server umount lustre-MDT0000 complete [ 6128.609454] LustreError: 379490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.618505] LustreError: 379490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6129.462341] LustreError: 379471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501585 with bad export cookie 9013030356661079989 [ 6129.468944] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6129.469810] LustreError: 379471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6135.722377] Lustre: server umount lustre-MDT0001 complete [ 6140.684342] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6142.302985] Key type lgssc unregistered [ 6142.482790] LNet: 384313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6142.487743] LNetError: 384313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6142.499744] LNet: Removed LNI 192.168.206.108@tcp [ 6142.996297] Key type .llcrypt unregistered [ 6142.998437] Key type ._llcrypt unregistered [ 6167.426517] Key type ._llcrypt registered [ 6167.428227] Key type .llcrypt registered [ 6167.494247] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6168.021334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6168.045647] alg: No test for adler32 (adler32-zlib) [ 6168.956807] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6169.067408] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6170.663162] Key type lgssc registered [ 6171.312815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6175.256598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6175.269179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.380745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6176.390371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6176.486247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.524459] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6182.745163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.800904] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6182.819856] Lustre: 387086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6182.991203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6184.991107] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6188.511240] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6191.210897] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6193.120408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6193.125319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6193.133971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.193780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.194982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.384164] Lustre: server umount lustre-MDT0000 complete [ 6200.200173] LustreError: 388680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501656 with bad export cookie 14686905005363165295 [ 6200.210783] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6200.358359] Lustre: server umount lustre-MDT0001 complete [ 6204.823987] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:14:20 (1787501660) [ 6208.035985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6208.319303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.208189] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6213.601456] LustreError: 389597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6213.610540] LustreError: 389597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6214.289966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.505918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.464507] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6220.000886] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6222.592424] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6225.730094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.830561] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6225.941093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6228.466606] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6231.879739] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6240.518613] Lustre: Permanently deactivating lustre-OST0000 [ 6240.526983] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6241.658221] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6242.787980] LustreError: 392437:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6242.794684] LustreError: 392437:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6242.799237] LustreError: 392437:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6243.562054] LustreError: 392485:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6243.566575] LustreError: 392485:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6250.600763] Lustre: server umount lustre-OST0000 complete [ 6251.865652] LustreError: 392736:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6251.870395] LustreError: 392736:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6255.584408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6255.588960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.597211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6255.600894] Lustre: Skipped 1 previous similar message [ 6257.633132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6257.634091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.753084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6262.756709] Lustre: Skipped 2 previous similar messages [ 6267.359166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6267.446568] Lustre: server umount lustre-MDT0000 complete [ 6267.873767] LustreError: 389597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.884082] LustreError: 389597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6269.179951] LustreError: 391427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501725 with bad export cookie 14686905005363165862 [ 6269.187092] LustreError: 391427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6269.187122] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6279.724938] Lustre: server umount lustre-MDT0001 complete [ 6283.761990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.637310] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6286.903549] LustreError: 393894:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6289.832463] Lustre: 394232:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.108@tcp [ 6290.826382] Lustre: server umount MGS complete [ 6296.153043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.371839] LustreError: 394955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.379180] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6296.384275] Lustre: Skipped 1 previous similar message [ 6296.417717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6298.143087] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6301.664999] LustreError: 394956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.134452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.273285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6304.015753] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6307.294849] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6309.847360] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6310.736222] Lustre: Permanently reactivating lustre-OST0000 [ 6310.745838] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6310.750144] Lustre: Skipped 2 previous similar messages [ 6332.383188] LustreError: 396592:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6336.768039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.928417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.932135] Lustre: Skipped 1 previous similar message [ 6339.462946] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6342.865314] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6352.357727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.358639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.366098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.371659] LustreError: Skipped 1 previous similar message [ 6356.069672] Lustre: server umount lustre-OST0000 complete [ 6358.496365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6362.594507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.602022] Lustre: Skipped 2 previous similar messages [ 6372.319168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6372.405908] Lustre: server umount lustre-MDT0000 complete [ 6372.833697] LustreError: 396256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.843042] LustreError: 396256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6374.114921] LustreError: 394937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501830 with bad export cookie 14686905005363166884 [ 6374.122119] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6374.122251] LustreError: 394937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6374.263473] Lustre: server umount lustre-MDT0001 complete [ 6378.889378] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6380.405295] Key type lgssc unregistered [ 6380.578715] LNet: 399418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6380.583496] LNetError: 399418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6380.595514] LNet: Removed LNI 192.168.206.108@tcp [ 6381.052993] Key type .llcrypt unregistered [ 6381.055191] Key type ._llcrypt unregistered [ 6388.961789] Key type ._llcrypt registered [ 6388.963218] Key type .llcrypt registered [ 6389.005158] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6396.861941] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6397.186855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6397.247110] alg: No test for adler32 (adler32-zlib) [ 6398.138865] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6398.265295] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6399.863167] Key type lgssc registered [ 6400.496252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6417.261128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6417.269973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6418.390113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6418.404722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6418.449727] Lustre: lustre-MDT0000: new disk, initializing [ 6418.490811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.501691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6420.340634] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6425.915765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.983530] Lustre: 402548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6426.024568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6426.027341] Lustre: Skipped 1 previous similar message [ 6426.062206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6426.074161] Lustre: lustre-MDT0001: new disk, initializing [ 6426.116184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6426.136180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6426.142813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6428.104285] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6433.047972] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6435.623051] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6438.912044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6439.090278] Lustre: lustre-OST0000: new disk, initializing [ 6439.093994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6439.099171] Lustre: 404513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6439.155547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.915349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6440.921417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6440.982456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6441.879880] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6446.754305] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.718321] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6448.861694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6450.771438] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6450.899825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6456.293913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6456.296124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.300750] LustreError: Skipped 1 previous similar message [ 6456.309491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6456.315930] Lustre: Skipped 1 previous similar message [ 6457.973761] Lustre: server umount lustre-OST0000 complete [ 6461.411498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.413060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.418911] Lustre: Skipped 2 previous similar messages [ 6466.529355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6466.533205] Lustre: Skipped 2 previous similar messages [ 6471.649536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6471.653771] Lustre: Skipped 1 previous similar message [ 6474.207279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6474.297643] Lustre: server umount lustre-MDT0000 complete [ 6476.089463] LustreError: 402541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501932 with bad export cookie 15583167929493247348 [ 6476.096760] LustreError: 402541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6476.097194] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6476.284738] Lustre: server umount lustre-MDT0001 complete [ 6480.264929] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6481.097358] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:18:56 (1787501936) [ 6484.315270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.560509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.379315] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6489.569249] LustreError: 406973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.578520] LustreError: 406973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6490.361733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.514072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.232713] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6495.639717] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6498.350758] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6501.591403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.780791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.361398] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6507.879658] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6508.841380] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6515.911925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6520.297714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6520.299279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.306607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6520.312410] Lustre: Skipped 1 previous similar message [ 6520.318552] Lustre: Skipped 1 previous similar message [ 6523.820529] Lustre: server umount lustre-OST0000 complete [ 6526.432388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.436033] LustreError: Skipped 1 previous similar message [ 6526.438768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.532449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6531.647686] Lustre: server umount lustre-MDT0000 complete [ 6533.459263] LustreError: 406954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787501989 with bad export cookie 15583167929493248230 [ 6533.466364] LustreError: 406954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6533.468306] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6533.630890] Lustre: server umount lustre-MDT0001 complete [ 6538.633337] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6540.273284] Key type lgssc unregistered [ 6540.438690] LNet: 410700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6540.443450] LNetError: 410700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6540.452928] LNet: Removed LNI 192.168.206.108@tcp [ 6540.961066] Key type .llcrypt unregistered [ 6540.963435] Key type ._llcrypt unregistered [ 6551.996532] Key type ._llcrypt registered [ 6551.998609] Key type .llcrypt registered [ 6552.067453] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6552.261536] Key type .llcrypt unregistered [ 6552.263848] Key type ._llcrypt unregistered [ 6561.661812] Key type ._llcrypt registered [ 6561.663678] Key type .llcrypt registered [ 6561.730644] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6562.071869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6562.083761] alg: No test for adler32 (adler32-zlib) [ 6562.998669] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6563.136504] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6564.743150] Key type lgssc registered [ 6565.375638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6569.351037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6569.360365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.595087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6572.623548] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6576.797094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6576.988296] LustreError: 412889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6576.995446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6576.999474] LustreError: 412889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6579.864735] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6580.066347] LustreError: 412894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.076080] LustreError: 412894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6580.078450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6583.501921] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6585.315443] LustreError: 412913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.322490] LustreError: 412913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6586.517982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.690653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6588.276970] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6597.090291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6597.095064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.097382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6597.102106] Lustre: Skipped 1 previous similar message [ 6597.111387] Lustre: Skipped 1 previous similar message [ 6602.213483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6602.217357] Lustre: Skipped 1 previous similar message [ 6602.536586] Lustre: server umount lustre-OST0000 complete [ 6606.819036] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6606.825951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.837859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.484499] Lustre: server umount lustre-MDT0000 complete [ 6612.226886] LustreError: 412874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787502068 with bad export cookie 3984868381260448783 [ 6612.233215] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6612.237823] LustreError: 412874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6612.386205] Lustre: server umount lustre-MDT0001 complete [ 6617.555655] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6619.043703] Key type lgssc unregistered [ 6619.203760] LNet: 416045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6619.210616] LNetError: 416045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6619.226576] LNet: Removed LNI 192.168.206.108@tcp [ 6619.691511] Key type .llcrypt unregistered [ 6619.693674] Key type ._llcrypt unregistered [ 6632.470338] Key type ._llcrypt registered [ 6632.472179] Key type .llcrypt registered [ 6632.535044] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6632.866916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6632.913858] alg: No test for adler32 (adler32-zlib) [ 6633.781949] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6633.895223] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6635.495154] Key type lgssc registered [ 6636.093790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6639.743996] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6639.755041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.199397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6660.319323] LustreError: 417421:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.108@tcp: failed processing log, type 4: rc = -110 [ 6686.943265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.680677] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6693.248127] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.495982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.691651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.728699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6698.595686] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6701.797920] LustreError: 419062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6701.807953] LustreError: 419062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6702.789322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.001689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6704.969964] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6708.659051] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6711.459732] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.408467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6717.413177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.421424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.435584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6718.438245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6718.445326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.487707] Lustre: server umount lustre-OST0000 complete [ 6723.555726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6723.556421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.557657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6723.557665] Lustre: Skipped 1 previous similar message [ 6723.580988] Lustre: Skipped 1 previous similar message [ 6728.505915] Lustre: server umount lustre-MDT0000 complete [ 6728.673304] LustreError: 418462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.682802] LustreError: 418462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6730.386561] LustreError: 418485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787502186 with bad export cookie 10541425276415810323 [ 6730.393545] LustreError: 418485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6730.393954] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6730.562712] Lustre: server umount lustre-MDT0001 complete [ 6735.801107] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6737.453302] Key type lgssc unregistered [ 6737.646849] LNet: 420994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6737.651454] LNetError: 420994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6737.662699] LNet: Removed LNI 192.168.206.108@tcp [ 6738.174782] Key type .llcrypt unregistered [ 6738.178049] Key type ._llcrypt unregistered [ 6750.613245] Key type ._llcrypt registered [ 6750.615083] Key type .llcrypt registered [ 6750.681698] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6751.059137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6751.085723] alg: No test for adler32 (adler32-zlib) [ 6751.957165] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6752.061526] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6753.655158] Key type lgssc registered [ 6754.276516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6758.164790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6758.176888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.397216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6761.461349] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6765.676343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.871785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6767.824274] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6771.439325] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6774.150936] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.463504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.682223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.494413] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6784.048544] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6784.880983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6795.916824] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.8@tcp (stopping) [ 6796.259518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.266853] Lustre: Skipped 2 previous similar messages [ 6798.013894] Lustre: server umount lustre-MDT0000 complete [ 6799.116371] LustreError: 424966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.8@tcp arrived at 1787502255 with bad export cookie 15621667441336583726 [ 6799.124465] LustreError: 424966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6800.064899] LustreError: 422366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787502256 with bad export cookie 15621667441336583110 [ 6800.071973] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6800.072432] LustreError: 422366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6801.377853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6801.378071] LustreError: 423577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6801.382378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.392311] LustreError: 423577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6801.400144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6801.408026] Lustre: Skipped 3 previous similar messages [ 6805.408483] LustreError: 422387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.409106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6805.417895] LustreError: 422387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6806.185629] Lustre: server umount lustre-OST0000 complete [ 6809.480939] Lustre: server umount lustre-MDT0001 complete [ 6814.477523] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6816.085671] Key type lgssc unregistered [ 6816.265712] LNet: 426074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6816.270872] LNetError: 426074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6816.281649] LNet: Removed LNI 192.168.206.108@tcp [ 6816.766159] Key type .llcrypt unregistered [ 6816.768325] Key type ._llcrypt unregistered [ 6829.049875] Key type ._llcrypt registered [ 6829.051447] Key type .llcrypt registered [ 6829.121809] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6829.573609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6829.605396] alg: No test for adler32 (adler32-zlib) [ 6830.519966] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6830.653729] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6832.263167] Key type lgssc registered [ 6832.887813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6836.845871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6836.857707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.065728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.053692] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6844.237608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.444381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6846.444334] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6849.982364] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6852.571697] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6855.807609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6855.982252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.800744] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6862.349239] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6863.155055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6874.080535] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6874.085296] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.093613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6874.597827] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.606987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6874.612712] Lustre: Skipped 1 previous similar message [ 6876.336878] Lustre: server umount lustre-MDT0001 complete [ 6879.714406] LustreError: 427486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6879.716524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.729053] LustreError: 427486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6879.737939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6884.837264] LustreError: 427467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.856497] Lustre: server umount lustre-OST0000 complete [ 6886.777984] Lustre: server umount lustre-MDT0000 complete [ 6893.253078] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 6894.844754] Key type lgssc unregistered [ 6895.034646] LNet: 431155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6895.039527] LNetError: 431155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6895.048675] LNet: Removed LNI 192.168.206.108@tcp [ 6895.556186] Key type .llcrypt unregistered [ 6895.558449] Key type ._llcrypt unregistered [ 6907.096824] Key type ._llcrypt registered [ 6907.098382] Key type .llcrypt registered [ 6907.165461] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 6915.338900] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6915.756607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6915.789118] alg: No test for adler32 (adler32-zlib) [ 6916.692578] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 6916.832918] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 6918.447148] Key type lgssc registered [ 6919.068715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6937.982566] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 6945.648445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6945.659158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.781547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6946.793306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6946.843322] Lustre: lustre-MDT0000: new disk, initializing [ 6946.877555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6946.886700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6948.842840] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6956.836420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.003355] Lustre: lustre-OST0000: new disk, initializing [ 6957.006563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6957.010182] Lustre: Skipped 1 previous similar message [ 6957.014575] Lustre: 436160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6957.059096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6959.813063] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6964.248144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6964.253773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6964.268449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6964.851053] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6972.619966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.700379] Lustre: 435143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6972.757897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6972.807476] Lustre: lustre-MDT0001: new disk, initializing [ 6972.861701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6972.878826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6972.881598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6974.822466] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 6979.707737] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6979.843604] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6981.880582] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6982.019084] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6984.430597] Lustre: server umount lustre-MDT0001 complete [ 6988.774830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.775346] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6993.893669] LustreError: 435148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.903070] LustreError: 435148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6999.012107] LustreError: 437497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.012276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6999.028749] Lustre: Skipped 1 previous similar message [ 6999.032777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.755594] Lustre: server umount lustre-OST0000 complete [ 7004.132084] LustreError: 435153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.889837] Lustre: server umount lustre-MDT0000 complete [ 7018.617931] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7020.214219] Key type lgssc unregistered [ 7020.393586] LNet: 439906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7020.398352] LNetError: 439906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7020.410638] LNet: Removed LNI 192.168.206.108@tcp [ 7020.910893] Key type .llcrypt unregistered [ 7020.914360] Key type ._llcrypt unregistered [ 7039.536852] Key type ._llcrypt registered [ 7039.538780] Key type .llcrypt registered [ 7039.604394] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7039.795564] Key type .llcrypt unregistered [ 7039.797658] Key type ._llcrypt unregistered [ 7048.462866] Key type ._llcrypt registered [ 7048.464052] Key type .llcrypt registered [ 7048.528844] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7049.027422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7049.044870] alg: No test for adler32 (adler32-zlib) [ 7049.945984] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7050.061782] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7051.671199] Key type lgssc registered [ 7052.292499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7065.672691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7065.682494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7066.804283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7066.819347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7066.862209] Lustre: lustre-MDT0000: new disk, initializing [ 7066.889025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.896783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7068.885351] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7074.519462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.594133] Lustre: 443261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7074.641995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7074.647666] Lustre: Skipped 1 previous similar message [ 7074.700520] Lustre: lustre-MDT0001: new disk, initializing [ 7074.754632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7074.772286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7074.777139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7076.726292] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7081.630668] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7084.197139] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7087.438870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.610582] Lustre: lustre-OST0000: new disk, initializing [ 7087.615370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7087.622416] Lustre: 445227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7087.680098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.497804] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7093.265418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7093.271985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7093.336262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7095.475861] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.340746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.344232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.348531] Lustre: Skipped 2 previous similar messages [ 7098.354153] Lustre: Skipped 1 previous similar message [ 7103.461156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.464914] Lustre: Skipped 3 previous similar messages [ 7104.442333] Lustre: server umount lustre-MDT0000 complete [ 7106.234371] LustreError: 444056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787502562 with bad export cookie 17170363082486443444 [ 7106.243173] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7106.401329] Lustre: server umount lustre-MDT0001 complete [ 7118.467112] Lustre: server umount lustre-OST0000 complete [ 7123.250671] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:29:38 (1787502578) [ 7126.976414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.250575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7129.042364] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7132.641885] LustreError: 447255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.653338] LustreError: 447255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7133.108956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7133.302373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.071970] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7138.478278] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7141.086989] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7157.347312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.459420] Lustre: Found index 0 for lustre-OST0000, updating log [ 7157.553869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7160.184887] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7163.575321] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.934288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7179.235627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7179.239226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.240273] LustreError: Skipped 1 previous similar message [ 7179.248743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7179.254214] Lustre: Skipped 1 previous similar message [ 7181.991830] Lustre: server umount lustre-OST0000 complete [ 7196.642282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7196.643344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7196.649854] Lustre: Skipped 2 previous similar messages [ 7196.655975] Lustre: Skipped 3 previous similar messages [ 7201.389457] Lustre: server umount lustre-MDT0000 complete [ 7201.761276] LustreError: 447254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.776111] LustreError: 447254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7203.128070] LustreError: 447235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787502659 with bad export cookie 17170363082486444823 [ 7203.134893] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7203.135139] LustreError: 447235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7203.272412] Lustre: server umount lustre-MDT0001 complete [ 7207.111552] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:31:02 (1787502662) [ 7224.159376] LNet: 1 local NIs in recovery (showing 1): 192.168.206.108@tcp [ 7793.857733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.086469] LustreError: 453678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7794.125254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.963223] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7799.265677] LustreError: 453679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7800.092567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.262170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7802.030791] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7805.532882] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7808.163884] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7824.421105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7824.574142] Lustre: Found index 0 for lustre-OST0000, updating log [ 7824.744849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.446747] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7830.915920] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7833.065360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7833.065691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.079323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7837.988667] Lustre: server umount lustre-OST0000 complete [ 7840.867293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.870473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.874871] Lustre: Skipped 1 previous similar message [ 7841.890385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7841.894776] Lustre: Skipped 2 previous similar messages [ 7843.938099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7843.942360] Lustre: Skipped 3 previous similar messages [ 7845.985302] LustreError: 453679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7845.993143] LustreError: 453679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7846.010554] Lustre: server umount lustre-MDT0000 complete [ 7847.800944] LustreError: 453660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787503303 with bad export cookie 17170363082486445908 [ 7847.807979] LustreError: 453660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7847.808960] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7847.999319] Lustre: server umount lustre-MDT0001 complete [ 7851.871449] LNet: 1 local NIs in recovery (showing 1): 192.168.206.108@tcp [ 7868.008820] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 7872.688035] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7890.204260] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:42:25 (1787503345) [ 7898.575877] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 7900.182562] Key type lgssc unregistered [ 7900.360664] LNet: 460334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7900.365464] LNetError: 460334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7955.685631] LNet: Removed LNI 192.168.206.108@tcp [ 7956.191139] Key type .llcrypt unregistered [ 7956.194068] Key type ._llcrypt unregistered [ 7964.968070] Key type ._llcrypt registered [ 7964.970120] Key type .llcrypt registered [ 7965.038284] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 7965.446746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7965.495394] alg: No test for adler32 (adler32-zlib) [ 7966.414740] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 7966.546894] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 7968.159196] Key type lgssc registered [ 7968.788612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7971.111733] LNet: Added LNI 192.168.206.108@tcp100 [8/256/0/180] [ 7971.486437] LNet: Added LNI 192.168.206.108@tcp101 [8/256/0/180] [ 7971.926564] LNet: Added LNI 192.168.206.108@tcp102 [8/256/0/180] [ 7972.358060] LNet: Added LNI 192.168.206.108@tcp103 [8/256/0/180] [ 7972.788458] LNet: Added LNI 192.168.206.108@tcp104 [8/256/0/180] [ 7973.220060] LNet: Added LNI 192.168.206.108@tcp105 [8/256/0/180] [ 7973.653806] LNet: Added LNI 192.168.206.108@tcp106 [8/256/0/180] [ 7974.091471] LNet: Added LNI 192.168.206.108@tcp107 [8/256/0/180] [ 7974.532261] LNet: Added LNI 192.168.206.108@tcp108 [8/256/0/180] [ 7974.966976] LNet: Added LNI 192.168.206.108@tcp109 [8/256/0/180] [ 7975.395057] LNet: Added LNI 192.168.206.108@tcp110 [8/256/0/180] [ 7975.824587] LNet: Added LNI 192.168.206.108@tcp111 [8/256/0/180] [ 7976.258035] LNet: Added LNI 192.168.206.108@tcp112 [8/256/0/180] [ 7976.693162] LNet: Added LNI 192.168.206.108@tcp113 [8/256/0/180] [ 7977.125931] LNet: Added LNI 192.168.206.108@tcp114 [8/256/0/180] [ 7977.545384] LNet: Added LNI 192.168.206.108@tcp115 [8/256/0/180] [ 7977.981733] LNet: Added LNI 192.168.206.108@tcp116 [8/256/0/180] [ 7978.415563] LNet: Added LNI 192.168.206.108@tcp117 [8/256/0/180] [ 7978.851924] LNet: Added LNI 192.168.206.108@tcp118 [8/256/0/180] [ 7979.269541] LNet: Added LNI 192.168.206.108@tcp119 [8/256/0/180] [ 7979.673327] LNet: Added LNI 192.168.206.108@tcp120 [8/256/0/180] [ 7980.104999] LNet: Added LNI 192.168.206.108@tcp121 [8/256/0/180] [ 7980.533940] LNet: Added LNI 192.168.206.108@tcp122 [8/256/0/180] [ 7980.957546] LNet: Added LNI 192.168.206.108@tcp123 [8/256/0/180] [ 7981.397859] LNet: Added LNI 192.168.206.108@tcp124 [8/256/0/180] [ 7981.827538] LNet: Added LNI 192.168.206.108@tcp125 [8/256/0/180] [ 7982.262930] LNet: Added LNI 192.168.206.108@tcp126 [8/256/0/180] [ 7982.690575] LNet: Added LNI 192.168.206.108@tcp127 [8/256/0/180] [ 7983.130685] LNet: Added LNI 192.168.206.108@tcp128 [8/256/0/180] [ 7983.572279] LNet: Added LNI 192.168.206.108@tcp129 [8/256/0/180] [ 7984.001713] LNet: Added LNI 192.168.206.108@tcp130 [8/256/0/180] [ 7984.432823] LNet: Added LNI 192.168.206.108@tcp131 [8/256/0/180] [ 7984.861460] LNet: Added LNI 192.168.206.108@tcp132 [8/256/0/180] [ 7985.290677] LNet: Added LNI 192.168.206.108@tcp133 [8/256/0/180] [ 7985.724668] LNet: Added LNI 192.168.206.108@tcp134 [8/256/0/180] [ 7986.159531] LNet: Added LNI 192.168.206.108@tcp135 [8/256/0/180] [ 7989.445212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7989.455823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.518126] Lustre: 463332:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [ 7990.567911] LNetError: 463358:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [ 7990.570442] LNetError: Protocol error connecting to 192.168.206.108@tcp117 at host 192.168.206.108:988: is it running a compatible version of Lustre? [ 7990.610064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7990.626289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7990.671199] Lustre: lustre-MDT0000: new disk, initializing [ 7990.708439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.718936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7991.587159] LNetError: 460796:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [ 7991.594288] LNetError: Protocol error connecting to 192.168.206.108@tcp112 at host 192.168.206.108:988: is it running a compatible version of Lustre? [ 7992.578558] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 7998.209936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.285033] Lustre: 463349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7998.324454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7998.328699] Lustre: Skipped 1 previous similar message [ 7998.378710] Lustre: lustre-MDT0001: new disk, initializing [ 7998.397514] Lustre: 464153:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [ 7998.403719] Lustre: 464153:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7998.479394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7998.504254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7998.510774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8000.444611] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8004.098523] Lustre: 464681:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [ 8004.105200] Lustre: 464681:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 8005.448270] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8008.131279] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8011.460486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8011.655396] Lustre: lustre-OST0000: new disk, initializing [ 8011.658960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8011.664321] Lustre: 465325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8011.724791] Lustre: 465330:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.108@tcp: can't add NID 192.168.206.108@tcp130: rc = -75 [ 8011.728213] Lustre: 465330:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 8011.752448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8014.171532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8014.176077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8014.246121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8014.508319] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8019.575207] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8023.904703] LNetError: 463352:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [ 8023.911204] LNetError: Protocol error connecting to 192.168.206.108@tcp117 at host 192.168.206.108:988: is it running a compatible version of Lustre? [ 8024.552193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.553364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.556850] LustreError: Skipped 1 previous similar message [ 8024.567113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.572029] Lustre: Skipped 1 previous similar message [ 8029.671163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.674881] Lustre: Skipped 1 previous similar message [ 8034.790420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8034.794143] Lustre: Skipped 1 previous similar message [ 8036.192605] LNetError: 463355:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [ 8036.198912] LNetError: 463355:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8036.203933] LNetError: Protocol error connecting to 192.168.206.108@tcp117 at host 192.168.206.108:988: is it running a compatible version of Lustre? [ 8036.210635] LNetError: Skipped 1 previous similar message [ 8037.343125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8037.434549] Lustre: server umount lustre-OST0000 complete [ 8039.392586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8039.397082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.404074] Lustre: Skipped 1 previous similar message [ 8039.407327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.025898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.029908] Lustre: Skipped 2 previous similar messages [ 8045.437146] Lustre: server umount lustre-MDT0000 complete [ 8047.275795] LustreError: 463342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787503503 with bad export cookie 188530530611847404 [ 8047.282796] LustreError: 463342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8047.284536] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8047.450977] Lustre: server umount lustre-MDT0001 complete [ 8048.480548] LNetError: 460794:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.108 [ 8048.486587] LNetError: 460794:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8048.491216] LNetError: Protocol error connecting to 192.168.206.108@tcp117 at host 192.168.206.108:988: is it running a compatible version of Lustre? [ 8048.497466] LNetError: Skipped 1 previous similar message [ 8052.598890] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8054.199495] Key type lgssc unregistered [ 8054.426730] LNet: 467241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8054.431972] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8054.436624] LNet: Removed LNI 192.168.206.108@tcp [ 8055.455277] LNet: Removed LNI 192.168.206.108@tcp100 [ 8055.457842] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8055.462326] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8055.467192] LNet: Removed LNI 192.168.206.108@tcp101 [ 8056.479213] LNet: Removed LNI 192.168.206.108@tcp102 [ 8056.481839] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8056.486301] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8056.491182] LNet: Removed LNI 192.168.206.108@tcp103 [ 8057.503177] LNet: Removed LNI 192.168.206.108@tcp104 [ 8058.527176] LNet: Removed LNI 192.168.206.108@tcp105 [ 8058.529863] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8058.534319] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8058.539191] LNet: Removed LNI 192.168.206.108@tcp106 [ 8059.551213] LNet: Removed LNI 192.168.206.108@tcp107 [ 8060.575222] LNet: Removed LNI 192.168.206.108@tcp108 [ 8061.599244] LNet: Removed LNI 192.168.206.108@tcp109 [ 8062.623225] LNet: Removed LNI 192.168.206.108@tcp110 [ 8062.625837] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8062.630267] LNetError: 467241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8062.635089] LNet: Removed LNI 192.168.206.108@tcp111 [ 8063.647178] LNet: Removed LNI 192.168.206.108@tcp112 [ 8064.671210] LNet: Removed LNI 192.168.206.108@tcp113 [ 8065.695186] LNet: Removed LNI 192.168.206.108@tcp114 [ 8066.719171] LNet: Removed LNI 192.168.206.108@tcp115 [ 8067.743226] LNet: Removed LNI 192.168.206.108@tcp116 [ 8068.767225] LNet: Removed LNI 192.168.206.108@tcp117 [ 8068.769847] LNet: Removed LNI 192.168.206.108@tcp118 [ 8068.772422] LNet: Removed LNI 192.168.206.108@tcp119 [ 8068.774936] LNet: Removed LNI 192.168.206.108@tcp120 [ 8068.777469] LNet: Removed LNI 192.168.206.108@tcp121 [ 8068.779952] LNet: Removed LNI 192.168.206.108@tcp122 [ 8068.782505] LNet: Removed LNI 192.168.206.108@tcp123 [ 8068.784932] LNet: Removed LNI 192.168.206.108@tcp124 [ 8068.787409] LNet: Removed LNI 192.168.206.108@tcp125 [ 8068.789874] LNet: Removed LNI 192.168.206.108@tcp126 [ 8068.792243] LNet: Removed LNI 192.168.206.108@tcp127 [ 8068.794661] LNet: Removed LNI 192.168.206.108@tcp128 [ 8068.797102] LNet: Removed LNI 192.168.206.108@tcp129 [ 8068.799639] LNet: Removed LNI 192.168.206.108@tcp130 [ 8068.802059] LNet: Removed LNI 192.168.206.108@tcp131 [ 8068.804518] LNet: Removed LNI 192.168.206.108@tcp132 [ 8068.806932] LNet: Removed LNI 192.168.206.108@tcp133 [ 8068.809415] LNet: Removed LNI 192.168.206.108@tcp134 [ 8068.818259] LNet: Removed LNI 192.168.206.108@tcp135 [ 8069.250085] Key type .llcrypt unregistered [ 8069.252520] Key type ._llcrypt unregistered [ 8079.630894] Key type ._llcrypt registered [ 8079.632913] Key type .llcrypt registered [ 8079.704820] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8087.915330] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8088.444972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8088.452909] alg: No test for adler32 (adler32-zlib) [ 8089.322114] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 8089.434131] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8091.047204] Key type lgssc registered [ 8091.666398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8108.714626] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:46:04 (1787503564) [ 8117.255870] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8118.829822] Key type lgssc unregistered [ 8119.004596] LNet: 471302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8119.009749] LNetError: 471302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8119.023610] LNet: Removed LNI 192.168.206.108@tcp [ 8119.504509] Key type .llcrypt unregistered [ 8119.507082] Key type ._llcrypt unregistered [ 8128.092770] Key type ._llcrypt registered [ 8128.094755] Key type .llcrypt registered [ 8128.162508] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8128.559917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8128.626051] alg: No test for adler32 (adler32-zlib) [ 8129.504488] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 8129.634869] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8131.247159] Key type lgssc registered [ 8131.822302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.535919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8136.545335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.661351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8137.676931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8137.730767] Lustre: lustre-MDT0000: new disk, initializing [ 8137.761939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.770423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8139.795907] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8145.368309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.443677] Lustre: 472493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8145.491046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8145.494667] Lustre: Skipped 1 previous similar message [ 8145.543956] Lustre: lustre-MDT0001: new disk, initializing [ 8145.598637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.620882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8145.628178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8147.588687] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8152.504559] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8155.181845] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8158.513645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.678693] Lustre: lustre-OST0000: new disk, initializing [ 8158.682159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8158.687465] Lustre: 474464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8158.742119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8161.488330] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8166.451177] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8166.928313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8166.935353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8166.992161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8168.192662] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing lnet_if_list [ 8170.493263] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8170.511874] LNet: Added LNI 192.168.206.108@tcp10 [8/256/0/180] [ 8173.258773] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8173.278239] LNet: Added LNI 192.168.206.108@tcp11 [8/256/0/180] [ 8175.304642] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8175.325840] LNet: Added LNI 192.168.206.108@tcp12 [8/256/0/180] [ 8177.402269] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8177.419199] LNet: Added LNI 192.168.206.108@tcp13 [8/256/0/180] [ 8179.545792] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8179.565789] LNet: Added LNI 192.168.206.108@tcp14 [8/256/0/180] [ 8181.792417] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8181.812506] LNet: Added LNI 192.168.206.108@tcp15 [8/256/0/180] [ 8184.097401] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8184.117665] LNet: Added LNI 192.168.206.108@tcp16 [8/256/0/180] [ 8186.344336] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8186.358240] LNet: Added LNI 192.168.206.108@tcp17 [8/256/0/180] [ 8188.657864] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8188.676851] LNet: Added LNI 192.168.206.108@tcp18 [8/256/0/180] [ 8191.125444] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8191.146719] LNet: Added LNI 192.168.206.108@tcp19 [8/256/0/180] [ 8193.701304] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8193.722514] LNet: Added LNI 192.168.206.108@tcp20 [8/256/0/180] [ 8196.341632] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8196.362603] LNet: Added LNI 192.168.206.108@tcp21 [8/256/0/180] [ 8198.968891] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8198.988785] LNet: Added LNI 192.168.206.108@tcp22 [8/256/0/180] [ 8201.608199] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8201.628437] LNet: Added LNI 192.168.206.108@tcp23 [8/256/0/180] [ 8204.431182] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8204.451289] LNet: Added LNI 192.168.206.108@tcp24 [8/256/0/180] [ 8207.261679] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8207.281134] LNet: Added LNI 192.168.206.108@tcp25 [8/256/0/180] [ 8210.068615] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8210.087974] LNet: Added LNI 192.168.206.108@tcp26 [8/256/0/180] [ 8212.888287] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8212.907956] LNet: Added LNI 192.168.206.108@tcp27 [8/256/0/180] [ 8215.615902] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8215.633266] LNet: Added LNI 192.168.206.108@tcp28 [8/256/0/180] [ 8218.284529] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8218.300247] LNet: Added LNI 192.168.206.108@tcp29 [8/256/0/180] [ 8221.129556] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8221.147807] LNet: Added LNI 192.168.206.108@tcp30 [8/256/0/180] [ 8223.933287] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8223.951817] LNet: Added LNI 192.168.206.108@tcp31 [8/256/0/180] [ 8226.994285] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8227.012595] LNet: Added LNI 192.168.206.108@tcp32 [8/256/0/180] [ 8229.974523] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8229.992217] LNet: Added LNI 192.168.206.108@tcp33 [8/256/0/180] [ 8232.931551] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8232.950287] LNet: Added LNI 192.168.206.108@tcp34 [8/256/0/180] [ 8236.050339] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8236.068459] LNet: Added LNI 192.168.206.108@tcp35 [8/256/0/180] [ 8239.089844] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8239.106844] LNet: Added LNI 192.168.206.108@tcp36 [8/256/0/180] [ 8242.300272] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8242.318848] LNet: Added LNI 192.168.206.108@tcp37 [8/256/0/180] [ 8245.514990] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8245.534662] LNet: Added LNI 192.168.206.108@tcp38 [8/256/0/180] [ 8248.678419] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8248.698252] LNet: Added LNI 192.168.206.108@tcp39 [8/256/0/180] [ 8251.816133] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8251.833425] LNet: Added LNI 192.168.206.108@tcp40 [8/256/0/180] [ 8255.411893] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8255.431871] LNet: Added LNI 192.168.206.108@tcp41 [8/256/0/180] [ 8259.165011] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8259.182426] LNet: Added LNI 192.168.206.108@tcp42 [8/256/0/180] [ 8262.970432] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8262.989861] LNet: Added LNI 192.168.206.108@tcp43 [8/256/0/180] [ 8266.996680] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8267.016294] LNet: Added LNI 192.168.206.108@tcp44 [8/256/0/180] [ 8270.938864] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8270.956057] LNet: Added LNI 192.168.206.108@tcp45 [8/256/0/180] [ 8274.706890] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8274.722602] LNet: Added LNI 192.168.206.108@tcp46 [8/256/0/180] [ 8278.279751] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8278.294906] LNet: Added LNI 192.168.206.108@tcp47 [8/256/0/180] [ 8281.881091] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8281.899572] LNet: Added LNI 192.168.206.108@tcp48 [8/256/0/180] [ 8285.540002] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8285.557247] LNet: Added LNI 192.168.206.108@tcp49 [8/256/0/180] [ 8289.471509] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8289.490033] LNet: Added LNI 192.168.206.108@tcp50 [8/256/0/180] [ 8299.488442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8299.492914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.500912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.002106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.005104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.009536] Lustre: Skipped 1 previous similar message [ 8300.015632] Lustre: Skipped 1 previous similar message [ 8305.123187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.126893] Lustre: Skipped 2 previous similar messages [ 8305.212875] Lustre: server umount lustre-MDT0000 complete [ 8306.964398] LustreError: 472485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787503762 with bad export cookie 14479974516761095207 [ 8306.970391] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8306.971315] LustreError: 472485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8307.124090] Lustre: server umount lustre-MDT0001 complete [ 8317.002714] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8321.105468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.623215] Lustre: 471766:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787503766/real 1787503766] req@000000008e956362 x1874333359243648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787503782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8326.638093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8327.647303] Lustre: 471767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787503767/real 1787503767] req@000000000b098ed5 x1874333359243776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787503783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8331.743301] Lustre: 471768:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787503771/real 1787503771] req@000000005c553b09 x1874333359243904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787503787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8331.743639] LustreError: 471764:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000031451fbb x1874333359244800/t0(0) o250->MGC192.168.206.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8331.931576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.968785] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8336.996544] LustreError: 498656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8337.005571] LustreError: 498656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8338.015259] Lustre: 471766:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787503777/real 1787503777] req@00000000e756e0a2 x1874333359244288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787503793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8338.030047] Lustre: 471766:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8338.215298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.408929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.245535] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8342.367130] Lustre: 471768:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787503782/real 1787503782] req@00000000db1a9f7b x1874333359244416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787503798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8343.531918] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8343.841608] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8346.512831] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8355.819847] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8357.856517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8357.861322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.869535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8358.508551] Lustre: server umount lustre-OST0000 complete [ 8366.050451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8366.054377] Lustre: Skipped 3 previous similar messages [ 8366.519182] Lustre: server umount lustre-MDT0000 complete [ 8368.319157] LustreError: 472484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787503824 with bad export cookie 14479974516761097356 [ 8368.326080] LustreError: 472484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8368.330063] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.931202] Lustre: server umount lustre-MDT0001 complete [ 8384.063634] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8385.418335] Key type lgssc unregistered [ 8385.608058] LNet: 501435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8385.612274] LNetError: 501435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8385.615879] LNet: Removed LNI 192.168.206.108@tcp [ 8385.617905] LNet: Removed LNI 192.168.206.108@tcp10 [ 8385.619735] LNet: Removed LNI 192.168.206.108@tcp11 [ 8385.621677] LNet: Removed LNI 192.168.206.108@tcp12 [ 8385.623508] LNet: Removed LNI 192.168.206.108@tcp13 [ 8385.625412] LNet: Removed LNI 192.168.206.108@tcp14 [ 8385.627269] LNet: Removed LNI 192.168.206.108@tcp15 [ 8385.629207] LNet: Removed LNI 192.168.206.108@tcp16 [ 8385.630955] LNet: Removed LNI 192.168.206.108@tcp17 [ 8385.632836] LNet: Removed LNI 192.168.206.108@tcp18 [ 8385.634627] LNet: Removed LNI 192.168.206.108@tcp19 [ 8385.636434] LNet: Removed LNI 192.168.206.108@tcp20 [ 8385.638082] LNet: Removed LNI 192.168.206.108@tcp21 [ 8385.639777] LNet: Removed LNI 192.168.206.108@tcp22 [ 8385.641526] LNet: Removed LNI 192.168.206.108@tcp23 [ 8385.643188] LNet: Removed LNI 192.168.206.108@tcp24 [ 8385.644864] LNet: Removed LNI 192.168.206.108@tcp25 [ 8385.646473] LNet: Removed LNI 192.168.206.108@tcp26 [ 8385.648211] LNet: Removed LNI 192.168.206.108@tcp27 [ 8385.650280] LNet: Removed LNI 192.168.206.108@tcp28 [ 8385.652422] LNet: Removed LNI 192.168.206.108@tcp29 [ 8385.654098] LNet: Removed LNI 192.168.206.108@tcp30 [ 8385.655774] LNet: Removed LNI 192.168.206.108@tcp31 [ 8385.657350] LNet: Removed LNI 192.168.206.108@tcp32 [ 8385.658967] LNet: Removed LNI 192.168.206.108@tcp33 [ 8385.660513] LNet: Removed LNI 192.168.206.108@tcp34 [ 8385.662037] LNet: Removed LNI 192.168.206.108@tcp35 [ 8385.663505] LNet: Removed LNI 192.168.206.108@tcp36 [ 8385.665003] LNet: Removed LNI 192.168.206.108@tcp37 [ 8385.666485] LNet: Removed LNI 192.168.206.108@tcp38 [ 8385.667980] LNet: Removed LNI 192.168.206.108@tcp39 [ 8385.669339] LNet: Removed LNI 192.168.206.108@tcp40 [ 8385.670812] LNet: Removed LNI 192.168.206.108@tcp41 [ 8385.672203] LNet: Removed LNI 192.168.206.108@tcp42 [ 8385.673646] LNet: Removed LNI 192.168.206.108@tcp43 [ 8385.675021] LNet: Removed LNI 192.168.206.108@tcp44 [ 8385.676325] LNet: Removed LNI 192.168.206.108@tcp45 [ 8385.677610] LNet: Removed LNI 192.168.206.108@tcp46 [ 8385.678867] LNet: Removed LNI 192.168.206.108@tcp47 [ 8385.680133] LNet: Removed LNI 192.168.206.108@tcp48 [ 8385.681338] LNet: Removed LNI 192.168.206.108@tcp49 [ 8385.686616] LNet: Removed LNI 192.168.206.108@tcp50 [ 8386.075546] Key type .llcrypt unregistered [ 8386.077754] Key type ._llcrypt unregistered [ 8395.108220] Key type ._llcrypt registered [ 8395.109312] Key type .llcrypt registered [ 8395.144909] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8403.200198] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8403.534216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8403.561399] alg: No test for adler32 (adler32-zlib) [ 8404.458659] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 8404.583439] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8406.199152] Key type lgssc registered [ 8406.809093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8423.686215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8423.696119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.809918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8424.826402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8424.877245] Lustre: lustre-MDT0000: new disk, initializing [ 8424.902302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.911614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8426.719951] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8432.200538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.265195] Lustre: 504605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8432.304629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8432.308315] Lustre: Skipped 1 previous similar message [ 8432.362651] Lustre: lustre-MDT0001: new disk, initializing [ 8432.414805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8432.436129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8432.442554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8434.266475] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8439.194786] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8441.867649] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8445.513555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.639482] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8445.680720] Lustre: lustre-OST0000: new disk, initializing [ 8445.683245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8445.686865] Lustre: 506623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8445.726781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.367351] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8453.133328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8453.139209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8453.196580] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8453.309291] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8458.211905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8458.214357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.216729] LustreError: Skipped 1 previous similar message [ 8458.225275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8458.230393] Lustre: Skipped 1 previous similar message [ 8458.857573] Lustre: server umount lustre-OST0000 complete [ 8463.330633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8463.331481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.332371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8463.346306] Lustre: Skipped 2 previous similar messages [ 8466.804053] Lustre: server umount lustre-MDT0000 complete [ 8468.561202] LustreError: 505400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787503924 with bad export cookie 17542097055347766239 [ 8468.566295] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8468.568157] LustreError: 505400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8468.724071] Lustre: server umount lustre-MDT0001 complete [ 8473.651533] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8475.115958] Key type lgssc unregistered [ 8475.267670] LNet: 508529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8475.272390] LNetError: 508529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8475.280565] LNet: Removed LNI 192.168.206.108@tcp [ 8475.710080] Key type .llcrypt unregistered [ 8475.712226] Key type ._llcrypt unregistered [ 8485.349467] Key type ._llcrypt registered [ 8485.350488] Key type .llcrypt registered [ 8485.387787] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8485.712751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8485.759479] alg: No test for adler32 (adler32-zlib) [ 8486.655882] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 8486.763245] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8488.367148] Key type lgssc registered [ 8488.986827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8492.825952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8492.836441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.061869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8496.029446] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8500.176080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.336426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8502.281951] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8505.884162] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8508.605741] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8511.896879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8512.070815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8514.662460] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8518.182581] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8533.528637] loop0: detected capacity change from 0 to 204800000 [ 8533.673162] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8533.680107] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8533.686963] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8536.700949] loop0: detected capacity change from 0 to 204800000 [ 8536.831820] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8536.838959] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8536.846414] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8539.406399] loop0: detected capacity change from 0 to 204800000 [ 8542.073782] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8542.157843] Lustre: fs15246-OST0000: new disk, initializing [ 8542.161067] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8542.165941] Lustre: 513388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8542.200312] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.974863] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8550.190746] loop1: detected capacity change from 0 to 204800000 [ 8552.809960] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.871106] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8552.913550] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8552.962117] Lustre: fs15246-MDT0000: new disk, initializing [ 8553.045940] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.056216] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8553.114257] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8554.144430] Lustre: 514510:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000211da9ee x1874333733523968/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787504026 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8554.938482] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8559.201775] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8559.208635] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8559.219720] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8569.325937] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.334966] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.502927] Lustre: server umount fs15246-MDT0000 complete [ 8574.441664] LustreError: 509773:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8574.451183] LustreError: 509773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8575.952418] Lustre: server umount fs15246-OST0000 complete [ 8581.088387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8581.093414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.101993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8582.113905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.118846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.127434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8584.674324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8587.056273] Lustre: server umount lustre-OST0000 complete [ 8589.795064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8589.796853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8589.803035] Lustre: Skipped 1 previous similar message [ 8589.806981] Lustre: Skipped 1 previous similar message [ 8595.068791] Lustre: server umount lustre-MDT0000 complete [ 8596.866658] LustreError: 515653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787504052 with bad export cookie 5432965447616946134 [ 8596.874203] LustreError: 515653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8596.874256] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8597.033861] Lustre: server umount lustre-MDT0001 complete [ 8602.314961] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8603.919728] Key type lgssc unregistered [ 8604.110758] LNet: 517610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8604.115820] LNetError: 517610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8604.125632] LNet: Removed LNI 192.168.206.108@tcp [ 8604.624081] Key type .llcrypt unregistered [ 8604.626326] Key type ._llcrypt unregistered [ 8614.584061] Key type ._llcrypt registered [ 8614.585722] Key type .llcrypt registered [ 8614.657359] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8615.091130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8615.149110] alg: No test for adler32 (adler32-zlib) [ 8616.057278] Lustre: Lustre: Build Version: 2.17.57_80_gd2e8623 [ 8616.178374] LNet: Added LNI 192.168.206.108@tcp [8/256/0/180] [ 8617.791157] Key type lgssc registered [ 8618.439794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8637.293270] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8641.869800] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8659.174570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8659.186175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.289212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8660.304597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8660.359121] Lustre: lustre-MDT0000: new disk, initializing [ 8660.386582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8660.396771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8662.200398] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8667.696253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.767598] Lustre: 522047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8667.809614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8667.813496] Lustre: Skipped 1 previous similar message [ 8667.860449] Lustre: lustre-MDT0001: new disk, initializing [ 8667.911758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8667.932830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8667.938936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8669.821414] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8674.772783] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8677.462708] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8680.753498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.916219] Lustre: lustre-OST0000: new disk, initializing [ 8680.935316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8680.940457] Lustre: 524012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8680.989455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.564126] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_default_debug -1 all [ 8688.518849] Lustre: DEBUG MARKER: oleg608-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8689.671098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8689.675717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8689.718347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8690.431367] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8690.550203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8692.473230] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8692.596644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8694.755371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8694.755688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8694.760526] LustreError: Skipped 1 previous similar message [ 8694.761559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8694.764042] Lustre: Skipped 1 previous similar message [ 8694.768037] Lustre: Skipped 1 previous similar message [ 8699.623575] Lustre: server umount lustre-OST0000 complete [ 8703.968693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8703.973768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.982605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8704.993128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8704.994211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.571607] Lustre: server umount lustre-MDT0000 complete [ 8709.346877] LustreError: 522039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787504165 with bad export cookie 5365379620811155683 [ 8709.353963] LustreError: 522039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8709.354249] LustreError: MGC192.168.206.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8709.502585] Lustre: server umount lustre-MDT0001 complete [ 8720.942831] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing set_hostid [ 8725.787581] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing load_modules_local [ 8741.853186] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8603 sec ========== 12:56:37 (1787504197) [ 8742.743739] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:56:38 (1787504198) === [ 8744.462350] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:56:39 (1787504199) === [ 8763.486362] Lustre: DEBUG MARKER: oleg608-server.virtnet: executing unload_modules_local [ 8764.965841] Key type lgssc unregistered [ 8765.141548] LNet: 530650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8765.146324] LNetError: 530650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8765.157543] LNet: Removed LNI 192.168.206.108@tcp [ 8765.612686] Key type .llcrypt unregistered [ 8765.614509] Key type ._llcrypt unregistered