[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482644110 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003304] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007734] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009005] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.014041] SELinux: Initializing. [ 0.015069] *** VALIDATE selinux *** [ 0.024482] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029844] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030193] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032140] *** VALIDATE tmpfs *** [ 0.034420] *** VALIDATE proc *** [ 0.035320] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037317] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040039] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043896] debug: unmapping init [mem 0xffffffff97e59000-0xffffffff97e60fff] [ 0.045156] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046762] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047041] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052021] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054379] rcu: Hierarchical SRCU implementation. [ 0.056698] smp: Bringing up secondary CPUs ... [ 0.057730] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.061287] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140450] node 0 deferred pages initialised in 74ms [ 0.143121] devtmpfs: initialized [ 0.144328] x86/mm: Memory block size: 128MB [ 0.148000] gcov: version magic: 0x41383552 [ 0.151314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157347] pinctrl core: initialized pinctrl subsystem [ 0.160210] [ 0.160927] ************************************************************* [ 0.163013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164023] ** ** [ 0.167012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169015] ** ** [ 0.171020] ** This means that this kernel is built to expose internal ** [ 0.174014] ** IOMMU data structures, which may compromise security on ** [ 0.176015] ** your system. ** [ 0.178017] ** ** [ 0.180019] ** If you see this message and you are not debugging the ** [ 0.183013] ** kernel, report this immediately to your vendor! ** [ 0.185025] ** ** [ 0.187010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188014] ************************************************************* [ 0.191183] NET: Registered protocol family 16 [ 0.193561] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202072] cpuidle: using governor menu [ 0.203875] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204451] PCI: Using configuration type 1 for base access [ 0.205143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219048] cryptd: max_cpu_qlen set to 1000 [ 0.221266] ACPI: Added _OSI(Module Device) [ 0.223014] ACPI: Added _OSI(Processor Device) [ 0.224013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230291] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235537] ACPI: Interpreter enabled [ 0.237064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238018] ACPI: Using IOAPIC for interrupt routing [ 0.240136] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.251744] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.253029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.256078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259037] acpiphp: Slot [2] registered [ 0.260068] acpiphp: Slot [5] registered [ 0.261092] acpiphp: Slot [6] registered [ 0.262097] acpiphp: Slot [7] registered [ 0.264135] acpiphp: Slot [8] registered [ 0.265131] acpiphp: Slot [9] registered [ 0.266086] acpiphp: Slot [10] registered [ 0.267126] acpiphp: Slot [3] registered [ 0.269098] acpiphp: Slot [4] registered [ 0.270091] acpiphp: Slot [11] registered [ 0.271057] acpiphp: Slot [12] registered [ 0.272089] acpiphp: Slot [13] registered [ 0.273091] acpiphp: Slot [14] registered [ 0.274105] acpiphp: Slot [15] registered [ 0.276144] acpiphp: Slot [16] registered [ 0.277073] acpiphp: Slot [17] registered [ 0.278143] acpiphp: Slot [18] registered [ 0.279123] acpiphp: Slot [19] registered [ 0.281142] acpiphp: Slot [20] registered [ 0.282121] acpiphp: Slot [21] registered [ 0.284117] acpiphp: Slot [22] registered [ 0.285114] acpiphp: Slot [23] registered [ 0.287079] acpiphp: Slot [24] registered [ 0.288220] acpiphp: Slot [25] registered [ 0.289092] acpiphp: Slot [26] registered [ 0.291075] acpiphp: Slot [27] registered [ 0.292112] acpiphp: Slot [28] registered [ 0.294121] acpiphp: Slot [29] registered [ 0.295137] acpiphp: Slot [30] registered [ 0.297149] acpiphp: Slot [31] registered [ 0.299072] PCI host bridge to bus 0000:00 [ 0.300023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.305037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.307022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.311021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313805] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316072] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.324745] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.329783] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.337529] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339598] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346877] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362796] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367742] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373748] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.415441] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.448017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460888] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.468019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.474014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.496017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.505183] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.513028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.538032] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.548858] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.557020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.564022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.586022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.601793] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.609025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.616027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.653033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670649] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673435] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676275] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677253] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679168] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683046] iommu: Default domain type: Passthrough [ 0.685347] SCSI subsystem initialized [ 0.686128] ACPI: bus type USB registered [ 0.687094] usbcore: registered new interface driver usbfs [ 0.689086] usbcore: registered new interface driver hub [ 0.690075] usbcore: registered new device driver usb [ 0.692199] pps_core: LinuxPPS API ver. 1 registered [ 0.694014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.697127] PTP clock support registered [ 0.699076] EDAC MC: Ver: 3.0.0 [ 0.701175] PCI: Using ACPI for IRQ routing [ 0.703027] NetLabel: Initializing [ 0.705011] NetLabel: domain hash size = 128 [ 0.706011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.709108] NetLabel: unlabeled traffic allowed by default [ 0.711171] vgaarb: loaded [ 0.713331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.715015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.720381] clocksource: Switched to clocksource kvm-clock [ 0.820769] VFS: Disk quotas dquot_6.6.0 [ 0.822432] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825110] *** VALIDATE ramfs *** [ 0.826425] *** VALIDATE hugetlbfs *** [ 0.827949] pnp: PnP ACPI init [ 0.831159] pnp: PnP ACPI: found 6 devices [ 0.848064] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.851665] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.853947] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.856228] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.858744] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.861424] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.864214] NET: Registered protocol family 2 [ 0.866596] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.871520] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.875170] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.880485] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.884218] TCP: Hash tables configured (established 65536 bind 65536) [ 0.887435] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.890070] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.892365] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895126] NET: Registered protocol family 1 [ 0.898537] RPC: Registered named UNIX socket transport module. [ 0.900939] RPC: Registered udp transport module. [ 0.903423] RPC: Registered tcp transport module. [ 0.905261] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.907721] NET: Registered protocol family 44 [ 0.908961] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.910722] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.912736] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.915299] PCI: CLS 0 bytes, default 64 [ 0.917452] Unpacking initramfs... [ 2.338259] debug: unmapping init [mem 0xffff905bbcc54000-0xffff905bbffbffff] [ 2.342557] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.344918] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.348164] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.841031] Initialise system trusted keyrings [ 2.842269] Key type blacklist registered [ 2.843810] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.854237] zbud: loaded [ 2.857588] *** VALIDATE nfs *** [ 2.858962] *** VALIDATE nfs4 *** [ 2.860943] pstore: using deflate compression [ 2.866288] Platform Keyring initialized [ 2.959590] NET: Registered protocol family 38 [ 2.961193] Key type asymmetric registered [ 2.962421] Asymmetric key parser 'x509' registered [ 2.964159] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.966454] io scheduler mq-deadline registered [ 2.967769] io scheduler kyber registered [ 2.969844] io scheduler bfq registered [ 2.971824] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.974227] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.976361] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.978835] ACPI: Power Button [PWRF] [ 2.983204] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.988633] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.997838] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.002944] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.014039] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.038319] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.063761] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.067765] Non-volatile memory driver v1.3 [ 3.068912] Linux agpgart interface v0.103 [ 3.097032] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.100266] vda: detected capacity change from 0 to 74702848 [ 3.113991] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.117472] vdb: detected capacity change from 0 to 1073741824 [ 3.131229] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.134326] vdc: detected capacity change from 0 to 2621440000 [ 3.147921] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.150316] vdd: detected capacity change from 0 to 2621440000 [ 3.163759] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.166123] vde: detected capacity change from 0 to 4294967296 [ 3.177659] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.179741] vdf: detected capacity change from 0 to 4294967296 [ 3.184664] libphy: Fixed MDIO Bus: probed [ 3.193492] usbcore: registered new interface driver usbserial_generic [ 3.195194] usbserial: USB Serial support registered for generic [ 3.196740] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.201407] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.203070] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.205125] mousedev: PS/2 mouse device common for all mice [ 3.207169] rtc_cmos 00:05: RTC can wake from S4 [ 3.210087] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.210127] rtc_cmos 00:05: registered as rtc0 [ 3.214111] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.216300] intel_pstate: CPU model not supported [ 3.218487] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.219187] hid: raw HID events driver (C) Jiri Kosina [ 3.223634] usbcore: registered new interface driver usbhid [ 3.225113] usbhid: USB HID core driver [ 3.226190] drop_monitor: Initializing network drop monitor service [ 3.227715] Initializing XFRM netlink socket [ 3.228955] NET: Registered protocol family 10 [ 3.229086] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.230892] Segment Routing with IPv6 [ 3.233186] NET: Registered protocol family 17 [ 3.235696] mpls_gso: MPLS GSO support [ 3.241248] RAS: Correctable Errors collector initialized. [ 3.243508] AVX version of gcm_enc/dec engaged. [ 3.244530] AES CTR mode by8 optimization enabled [ 3.307910] sched_clock: Marking stable (3307887397, 0)->(4181096247, -873208850) [ 3.312034] registered taskstats version 1 [ 3.314628] Loading compiled-in X.509 certificates [ 3.316231] zswap: loaded using pool lzo/zbud [ 3.335923] Key type big_key registered [ 3.345958] Key type encrypted registered [ 3.347519] ima: No TPM chip found, activating TPM-bypass! [ 3.349562] ima: Allocated hash algorithm: sha1 [ 3.351230] ima: No architecture policies found [ 3.352585] evm: Initialising EVM extended attributes: [ 3.353889] evm: security.selinux [ 3.354767] evm: security.ima [ 3.355843] evm: security.capability [ 3.357061] evm: HMAC attrs: 0x1 [ 3.359179] rtc_cmos 00:05: setting system clock to 2026-08-15 20:28:25 UTC (1786825705) [ 3.365140] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.367883] debug: unmapping init [mem 0xffffffff97b82000-0xffffffff97e58fff] [ 3.380073] Write protecting the kernel read-only data: 28672k [ 3.382855] debug: unmapping init [mem 0xffffffff96203000-0xffffffff963fffff] [ 3.384843] debug: unmapping init [mem 0xffffffff96b14000-0xffffffff96bfffff] [ 3.415440] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.421251] systemd[1]: Detected virtualization kvm. [ 3.422612] systemd[1]: Detected architecture x86-64. [ 3.424224] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.452147] systemd[1]: No hostname configured. [ 3.453283] systemd[1]: Set hostname to . [ 3.454702] random: systemd: uninitialized urandom read (16 bytes read) [ 3.456303] systemd[1]: Initializing machine ID from random generator. [ 3.498161] random: ln: uninitialized urandom read (6 bytes read) [ 3.583417] random: systemd: uninitialized urandom read (16 bytes read) [ 3.584949] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.588031] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.591530] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.144347] device-mapper: uevent: version 1.0.3 [ 4.146784] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.920900] random: fast init done [ 4.945977] virtio_net virtio0 ens2: renamed from eth0 [ 4.982039] scsi host0: ata_piix [ 5.013737] scsi host1: ata_piix [ 5.015046] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.017642] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.725838] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.861639] random: crng init done [ 9.863271] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.301788] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.490259] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.736110] SELinux: Disabled at runtime. [ 11.794048] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.804172] systemd[1]: Detected virtualization kvm. [ 11.806373] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.277894] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.283566] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.293298] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.297886] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.300943] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.307781] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.311226] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ 12.353890] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all De[ 12.824584] squashfs: version 4.0 (2009/01/31) Phillip Lougher vices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.226676] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.239220] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.419862] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.432143] EDAC sbridge: Ver: 1.1.2 [ 15.059259] Key type dns_resolver registered [ 15.372898] NFS: Registering the id_resolver key type [ 15.375136] Key type id_resolver registered [ 15.376786] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg111-server login: [ 30.658418] hrtimer: interrupt took 11451619 ns [ 72.944897] libcfs: loading out-of-tree module taints kernel. [ 73.085964] Key type ._llcrypt registered [ 73.089809] Key type .llcrypt registered [ 73.362636] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 96.022962] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 98.412731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.425280] alg: No test for adler32 (adler32-zlib) [ 100.436903] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 101.797782] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 103.679232] Key type lgssc registered [ 106.146189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.817646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 167.072913] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 180.220786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.242716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.517783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.566229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.706596] Lustre: lustre-MDT0000: new disk, initializing [ 181.785604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.804072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.260858] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.225084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.344135] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.406146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.409764] Lustre: Skipped 1 previous similar message [ 200.504496] Lustre: lustre-MDT0001: new disk, initializing [ 200.563544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.590209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.606865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.406370] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.865613] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 216.932419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.174878] Lustre: lustre-OST0000: new disk, initializing [ 217.177875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.183150] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.244366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.610655] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.771336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.779273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.816253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.491362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.661826] Lustre: lustre-OST0001: new disk, initializing [ 233.664891] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 233.670727] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.719214] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.572427] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.751236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.756954] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.822544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 250.549403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.875776] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.874871] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.576604] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing yml_node [ 274.101684] Lustre: DEBUG MARKER: Client: 2.17.57.5 [ 276.129829] Lustre: DEBUG MARKER: MDS: 2.17.57.5 [ 278.420278] Lustre: DEBUG MARKER: OSS: 2.17.57.5 [ 279.780343] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 16:33:00 EDT 2026 [ 295.295984] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.717312] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.320202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.331528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.347052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.852736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.854461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.865589] Lustre: Skipped 2 previous similar messages [ 314.872019] Lustre: Skipped 3 previous similar messages [ 317.999362] Lustre: server umount lustre-MDT0000 complete [ 321.548762] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826023 with bad export cookie 8900373831798024907 [ 321.549938] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.555483] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 325.088097] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.089847] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.095589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.102741] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 325.119337] Lustre: Skipped 1 previous similar message [ 326.114960] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.138937] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 327.919891] Lustre: server umount lustre-MDT0001 complete [ 331.686836] Lustre: server umount lustre-OST0000 complete [ 335.537770] Lustre: server umount lustre-OST0001 complete [ 343.425611] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 354.621269] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 389.256752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.460828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.486520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 389.560706] Lustre: lustre-MDT0000: new disk, initializing [ 389.633872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.644808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 393.501396] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 402.648957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.773517] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.823929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.826225] Lustre: Skipped 1 previous similar message [ 402.903171] Lustre: lustre-MDT0001: new disk, initializing [ 402.980666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.019382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 403.029712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.758584] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 415.245595] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 420.059979] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 425.614989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.862954] Lustre: lustre-OST0000: new disk, initializing [ 425.865666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.871268] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.957435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.846535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 427.857337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.974832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.554906] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 438.805856] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 442.608457] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 442.804980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 447.158154] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 447.403987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 453.603556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.605522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 453.609030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.609038] Lustre: Skipped 3 previous similar messages [ 453.615805] Lustre: Skipped 1 previous similar message [ 453.638418] LustreError: Skipped 1 previous similar message [ 455.455628] Lustre: server umount lustre-OST0000 complete [ 459.233956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.243922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.253637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.258367] Lustre: Skipped 1 previous similar message [ 468.964197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 468.971628] Lustre: Skipped 3 previous similar messages [ 473.055369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 473.191403] Lustre: server umount lustre-MDT0000 complete [ 476.727035] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826178 with bad export cookie 8900373831798026440 [ 476.730762] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.738018] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 476.969560] Lustre: server umount lustre-MDT0001 complete [ 481.897652] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:36:21 (1786826181) [ 483.947179] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 486.208643] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:36:26 (1786826186) [ 494.063452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.567609] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.666120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.655943] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 499.682026] LustreError: 19452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.803371] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.046813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.149107] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 518.548493] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.893708] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 530.593788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 537.177317] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 545.248154] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 552.930478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.939209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.953980] Lustre: Skipped 1 previous similar message [ 552.959522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 552.964252] Lustre: Skipped 2 previous similar messages [ 559.226800] Lustre: server umount lustre-OST0000 complete [ 563.687280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.694829] LustreError: Skipped 1 previous similar message [ 569.043537] Lustre: server umount lustre-MDT0000 complete [ 570.339089] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.360159] LustreError: 19447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 573.025284] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826275 with bad export cookie 8900373831798027322 [ 573.033919] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.046769] LustreError: 19431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 573.474240] Lustre: server umount lustre-MDT0001 complete [ 586.159590] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 589.791685] Key type lgssc unregistered [ 590.143295] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.155546] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.208230] LNet: Removed LNI 192.168.201.111@tcp [ 592.386284] Key type .llcrypt unregistered [ 592.388565] Key type ._llcrypt unregistered [ 615.809118] Key type ._llcrypt registered [ 615.813845] Key type .llcrypt registered [ 615.937877] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 616.769394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.780290] alg: No test for adler32 (adler32-zlib) [ 617.893624] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 618.110887] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 619.775231] Key type lgssc registered [ 620.846355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.656784] loop: module loaded [ 628.700113] loop0: detected capacity change from 0 to 204800000 [ 628.808131] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.827211] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 629.170772] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.868861] loop0: detected capacity change from 0 to 204800000 [ 635.031178] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.050613] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 635.066327] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.574397] loop0: detected capacity change from 0 to 204800000 [ 644.257428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 644.285285] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.492693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 645.554248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 645.638845] Lustre: lustre-MDT0000: new disk, initializing [ 645.707605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.725581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 650.005588] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 658.498935] loop1: detected capacity change from 0 to 204800000 [ 663.720161] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.043469] Lustre: lustre-OST07c6: new disk, initializing [ 664.047984] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 664.054315] Lustre: Skipped 1 previous similar message [ 664.058313] Lustre: 26170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 664.122428] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 670.312427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 670.321175] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 670.365681] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 670.913412] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 676.560341] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 676.576382] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 676.584068] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 676.593195] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 676.608760] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 676.616381] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 698.698419] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 698.712939] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.721135] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.730939] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 698.737757] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.745209] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.951695] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 703.959254] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.963186] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 703.967861] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.975863] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.985848] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.993305] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 703.997889] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.002714] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 704.007852] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.012508] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.016777] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.327270] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 709.339297] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 709.354994] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 709.364646] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.374472] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.383471] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.389401] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 709.402955] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.418403] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 709.432239] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.438888] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.450432] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.071124] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 718.077453] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 718.081317] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 718.087072] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.092719] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.095467] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.098050] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 718.102000] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.106376] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.109984] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 718.112832] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.115220] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 728.821532] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 728.828249] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 728.838220] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 728.844906] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.859558] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.868483] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.875923] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 728.887338] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.896077] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 728.903336] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 728.909485] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.918250] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 782.819116] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 782.828497] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.852566] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 786.672569] Lustre: server umount lustre-OST07c6 complete [ 798.939466] Lustre: server umount lustre-MDT0000 complete [ 818.143429] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 821.018906] Key type lgssc unregistered [ 821.344321] LNet: 28730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.350347] LNetError: 28730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.370718] LNet: Removed LNI 192.168.201.111@tcp [ 822.363858] Key type .llcrypt unregistered [ 822.367922] Key type ._llcrypt unregistered [ 842.820667] Key type ._llcrypt registered [ 842.822827] Key type .llcrypt registered [ 842.948227] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 844.498944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.519920] alg: No test for adler32 (adler32-zlib) [ 845.773454] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 846.117434] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 847.855795] Key type lgssc registered [ 849.010735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 857.522212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.539611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.988062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.151608] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 869.346842] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.368864] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 871.660516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.198465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.933279] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 884.173994] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 890.553239] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 897.621858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.988935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.729747] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 904.177348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 912.244076] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 929.771311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 929.781567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.806659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.837464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.843646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.847920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.885872] Lustre: Skipped 1 previous similar message [ 935.820270] Lustre: server umount lustre-OST0000 complete [ 943.073529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.079852] Lustre: Skipped 1 previous similar message [ 943.083526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.092876] Lustre: Skipped 1 previous similar message [ 945.648802] Lustre: server umount lustre-MDT0000 complete [ 948.194697] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.210716] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 949.070174] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826651 with bad export cookie 3144205354536262880 [ 949.077037] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.096931] LustreError: 29982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 949.528476] Lustre: server umount lustre-MDT0001 complete [ 959.767089] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 962.523847] Key type lgssc unregistered [ 962.833320] LNet: 33742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.839735] LNetError: 33742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.858407] LNet: Removed LNI 192.168.201.111@tcp [ 963.787679] Key type .llcrypt unregistered [ 963.791193] Key type ._llcrypt unregistered [ 985.531411] Key type ._llcrypt registered [ 985.535159] Key type .llcrypt registered [ 985.662330] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 986.727752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.750119] alg: No test for adler32 (adler32-zlib) [ 987.927197] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 988.161845] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 989.847492] Key type lgssc registered [ 991.447327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1000.426610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1000.443125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1002.071769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.849630] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1012.195915] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.221861] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1016.086325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.614072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.530306] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1029.672521] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1036.046909] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1042.915459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.185752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.310947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1050.437905] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1059.493837] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1072.102412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.108639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1072.113842] Lustre: Skipped 1 previous similar message [ 1072.116495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1072.119707] Lustre: Skipped 1 previous similar message [ 1077.219556] LustreError: 36836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.263840] LustreError: 36836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1077.420079] Lustre: server umount lustre-OST0000 complete [ 1082.347312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.348891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.366684] Lustre: Skipped 1 previous similar message [ 1087.462916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.475657] Lustre: Skipped 2 previous similar messages [ 1089.768909] Lustre: server umount lustre-MDT0000 complete [ 1092.577698] LustreError: 35742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.597526] LustreError: 35742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1094.391363] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826796 with bad export cookie 11691021705532651585 [ 1094.402347] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.410730] LustreError: 34994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1095.282759] Lustre: server umount lustre-MDT0001 complete [ 1109.345604] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1113.579118] Key type lgssc unregistered [ 1114.033822] LNet: 38552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.039384] LNetError: 38552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.050325] LNet: Removed LNI 192.168.201.111@tcp [ 1115.225833] Key type .llcrypt unregistered [ 1115.230639] Key type ._llcrypt unregistered [ 1148.793944] Key type ._llcrypt registered [ 1148.798049] Key type .llcrypt registered [ 1148.970191] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1150.747803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.854252] alg: No test for adler32 (adler32-zlib) [ 1151.853729] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1151.991879] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1153.623161] Key type lgssc registered [ 1154.711994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.070166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1164.108917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.632298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.666968] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1175.010155] LustreError: 39972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.047307] LustreError: 39972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1180.131625] LustreError: 39971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.913117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.216419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.882971] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1194.437826] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1202.506995] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1210.911416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1211.369246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.303849] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1227.993871] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1234.941699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1247.712439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1247.721132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.741741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.552810] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1250.275667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.635273] Lustre: server umount lustre-MDT0000 complete [ 1254.630592] LustreError: 39967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.716100] LustreError: 39951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826958 with bad export cookie 3665628423719460687 [ 1256.723732] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.733831] LustreError: 39951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1259.759499] LustreError: 39966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.805850] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [ 1259.810571] LustreError: 39966:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1259.841126] Lustre: Skipped 3 previous similar messages [ 1260.513515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.523640] Lustre: Skipped 1 previous similar message [ 1264.874470] LustreError: 40978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.877027] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [ 1264.905997] LustreError: 40978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1264.948283] Lustre: Skipped 2 previous similar messages [ 1269.976434] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [ 1269.983841] Lustre: Skipped 1 previous similar message [ 1271.264116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1271.776233] Lustre: server umount lustre-MDT0001 complete [ 1287.575135] Lustre: server umount lustre-OST0000 complete [ 1308.636666] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1312.948690] Key type lgssc unregistered [ 1313.503562] LNet: 43862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.517873] LNetError: 43862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.535564] LNet: Removed LNI 192.168.201.111@tcp [ 1316.244271] Key type .llcrypt unregistered [ 1316.245827] Key type ._llcrypt unregistered [ 1344.601586] Key type ._llcrypt registered [ 1344.607984] Key type .llcrypt registered [ 1344.715718] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1345.831480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.864058] alg: No test for adler32 (adler32-zlib) [ 1346.989794] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1347.195103] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1348.871123] Key type lgssc registered [ 1350.047952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1357.781835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1357.808884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.126879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.213804] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1369.576159] LustreError: 45297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.608956] LustreError: 45297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1374.713460] LustreError: 45297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.882522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.544062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1382.874807] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1391.611882] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1398.574667] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1406.592619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1406.887228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.526798] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1422.279747] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1429.449079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1439.723633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.731552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1439.752994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1441.413582] Lustre: server umount lustre-OST0000 complete [ 1464.300846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.303332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1464.320902] Lustre: Skipped 2 previous similar messages [ 1464.321402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.321406] Lustre: Skipped 1 previous similar message [ 1466.420655] Lustre: server umount lustre-MDT0000 complete [ 1469.409843] LustreError: 46588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.442859] LustreError: 46588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1471.489048] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827173 with bad export cookie 4250445576690311830 [ 1471.501460] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.512694] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1478.082630] Lustre: server umount lustre-MDT0001 complete [ 1490.625465] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1493.584736] Key type lgssc unregistered [ 1493.905302] LNet: 49033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.919118] LNetError: 49033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.961265] LNet: Removed LNI 192.168.201.111@tcp [ 1494.741209] Key type .llcrypt unregistered [ 1494.743334] Key type ._llcrypt unregistered [ 1520.972906] Key type ._llcrypt registered [ 1520.976371] Key type .llcrypt registered [ 1521.093926] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1522.709376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.791806] alg: No test for adler32 (adler32-zlib) [ 1524.043549] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1524.391675] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1526.175613] Key type lgssc registered [ 1527.588718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.908488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.931713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.693167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.580369] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1548.256954] LustreError: 50450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.269499] LustreError: 50450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1553.387493] LustreError: 50451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.481881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1553.946575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.851095] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1567.880636] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1574.812653] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1583.937423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1584.372085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.920459] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1600.952749] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1607.939283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1617.841412] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1622.340613] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1634.952198] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1516h55m34s [ 1636.816829] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1640.928235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1640.932836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.941374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.022507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.028803] Lustre: Skipped 4 previous similar messages [ 1643.270614] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 1646.168981] Lustre: server umount lustre-MDT0000 complete [ 1648.135047] LustreError: 51439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.168053] LustreError: 51439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1650.408364] LustreError: 51439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.756782] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1656.538254] LustreError: 51439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.580142] LustreError: 51439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1659.014774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.326260] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.644335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.700048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1660.664583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1663.918782] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1668.328135] Lustre: DEBUG MARKER: Wait for df (36720) ... [ 1670.694356] Lustre: DEBUG MARKER: done [ 1676.258866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.265502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1676.283189] Lustre: Skipped 3 previous similar messages [ 1676.293302] Lustre: Skipped 2 previous similar messages [ 1681.380587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.392635] Lustre: Skipped 1 previous similar message [ 1682.528893] Lustre: server umount lustre-OST0000 complete [ 1686.513363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.699848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.709709] Lustre: Skipped 4 previous similar messages [ 1692.965461] Lustre: server umount lustre-MDT0000 complete [ 1693.793488] LustreError: 50445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.266863] LustreError: 50432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827400 with bad export cookie 9560081867194814934 [ 1698.273498] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1698.283076] LustreError: 50432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1699.004689] Lustre: server umount lustre-MDT0001 complete [ 1714.654736] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1718.409472] Key type lgssc unregistered [ 1718.966590] LNet: 55408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.978457] LNetError: 55408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.037832] LNet: Removed LNI 192.168.201.111@tcp [ 1721.398378] Key type .llcrypt unregistered [ 1721.406623] Key type ._llcrypt unregistered [ 1770.385549] Key type ._llcrypt registered [ 1770.387715] Key type .llcrypt registered [ 1770.525056] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1771.811982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.934474] alg: No test for adler32 (adler32-zlib) [ 1773.255528] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 1773.668636] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1775.407696] Key type lgssc registered [ 1777.081858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.911908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.946290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.234462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1789.247288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1789.348568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.485425] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1805.935070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.188953] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1806.230971] Lustre: 57622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1806.643799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1812.199553] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1822.005527] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1829.536538] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1837.166271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.271457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1837.477205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.517197] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1845.463548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1852.994211] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1861.016741] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1516h59m21s [ 1864.106350] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1873.049409] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1876.050147] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1876.099599] LustreError: 57628:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1896.218521] Lustre: lustre-MDT0000: Client 7ae69315-2be5-42c9-923c-45fbd745956c (at 192.168.201.11@tcp) reconnecting [ 1916.112219] LustreError: 57628:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1918.636272] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1921.874115] Lustre: DEBUG MARKER: done [ 1932.769658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.779617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.785903] LustreError: Skipped 1 previous similar message [ 1932.786838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.824333] Lustre: Skipped 1 previous similar message [ 1934.700937] Lustre: server umount lustre-OST0000 complete [ 1939.936665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1939.948831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.979281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.986777] Lustre: Skipped 1 previous similar message [ 1941.036145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1941.048868] Lustre: Skipped 2 previous similar messages [ 1943.074670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.081969] Lustre: Skipped 1 previous similar message [ 1945.950977] Lustre: server umount lustre-MDT0000 complete [ 1950.120771] LustreError: 57614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827652 with bad export cookie 2362721593082842994 [ 1950.141345] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.147622] LustreError: 57614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1950.935739] Lustre: server umount lustre-MDT0001 complete [ 1966.601957] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1970.680379] Key type lgssc unregistered [ 1971.270169] LNet: 61973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.277469] LNetError: 61973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.327942] LNet: Removed LNI 192.168.201.111@tcp [ 1973.928424] Key type .llcrypt unregistered [ 1973.929885] Key type ._llcrypt unregistered [ 2032.025508] Key type ._llcrypt registered [ 2032.027605] Key type .llcrypt registered [ 2032.115428] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2033.616393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2033.684930] alg: No test for adler32 (adler32-zlib) [ 2034.893210] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2035.233697] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2036.991247] Key type lgssc registered [ 2038.781951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2043.311600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2043.340382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.792526] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2044.860043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2045.176622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.295908] Lustre: Failing over lustre-MDT0000 [ 2046.977290] Lustre: server umount lustre-MDT0000 complete [ 2055.833431] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:02:35 (1786827755) [ 2063.421578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.895371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.075864] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2078.343270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.531386] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2078.609746] Lustre: 64776:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.898155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.818953] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2092.790210] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.618398] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2107.851896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2108.186417] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2108.382097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.911314] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2116.266049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2123.104330] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2130.261114] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2140.642697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2140.652175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.673570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.179178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.180306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2142.211429] Lustre: Skipped 1 previous similar message [ 2142.228382] Lustre: Skipped 2 previous similar messages [ 2145.739811] Lustre: server umount lustre-MDT0000 complete [ 2150.535435] LustreError: 64770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827852 with bad export cookie 18260956647960049038 [ 2150.543265] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.552728] LustreError: 64770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2150.913687] Lustre: server umount lustre-MDT0001 complete [ 2154.962885] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2160.928885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2168.800142] Lustre: 63699:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827854/real 1786827854] req@0000000060e406ec x1873624681195392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.825724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.972852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.871169] Lustre: 63698:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827857/real 1786827857] req@00000000b5ad910a x1873624681195520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2172.895183] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827859/real 1786827859] req@000000006a457e3f x1873624681195648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.313948] LustreError: 66631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.326359] LustreError: 66631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2175.412698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.529492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2176.415515] Lustre: 63700:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827862/real 1786827862] req@0000000038a314d6 x1873624681195904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2176.425305] LustreError: 66632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.482082] LustreError: 68554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.519394] LustreError: 68554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2180.313266] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2181.543263] Lustre: 63699:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827867/real 1786827867] req@0000000095b3b7f1 x1873624681196032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2183.653737] LustreError: 68552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.686689] LustreError: 68552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2188.769697] LustreError: 68554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.793406] LustreError: 68554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2190.412544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.887752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2195.950309] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2196.111883] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2197.056547] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2204.266691] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2211.004514] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2221.542302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2221.551822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.567230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.941318] Lustre: server umount lustre-MDT0000 complete [ 2227.689408] LustreError: 68552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.710626] LustreError: 68552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2228.062224] LustreError: 64771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786827930 with bad export cookie 18260956647960051320 [ 2228.064882] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2228.070084] LustreError: 64771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2228.385692] Lustre: server umount lustre-MDT0001 complete [ 2235.562587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2246.750695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.159706] Lustre: 63697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786827934/real 1786827934] req@00000000fb7bd658 x1873624681225472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786827950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.212234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.232476] Lustre: Skipped 2 previous similar messages [ 2253.645960] LustreError: 69287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.722643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.775471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2258.204372] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2268.266099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.200601] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2274.285836] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2277.353421] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2281.707400] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2288.031852] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2297.829126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.837671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.854776] Lustre: Skipped 2 previous similar messages [ 2297.887505] Lustre: Skipped 5 previous similar messages [ 2301.325123] Lustre: server umount lustre-MDT0000 complete [ 2302.951136] LustreError: 71237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.961522] LustreError: 71237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2305.712275] LustreError: 72861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828007 with bad export cookie 18260956647960052699 [ 2305.726726] LustreError: 72861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2305.728036] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.307300] Lustre: server umount lustre-MDT0001 complete [ 2313.741750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2317.787831] Lustre: DEBUG MARKER: files compared the same [ 2330.470595] Lustre: server umount lustre-OST0000 complete [ 2351.376844] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2355.030422] Key type lgssc unregistered [ 2355.369906] LNet: 74518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.387484] LNetError: 74518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.461511] LNet: Removed LNI 192.168.201.111@tcp [ 2357.267540] Key type .llcrypt unregistered [ 2357.269272] Key type ._llcrypt unregistered [ 2386.110800] Key type ._llcrypt registered [ 2386.114957] Key type .llcrypt registered [ 2386.273293] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2388.197550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.224056] alg: No test for adler32 (adler32-zlib) [ 2389.467987] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2389.904151] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2391.858865] Key type lgssc registered [ 2393.855014] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2404.547096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2404.568211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.367919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.385912] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug +malloc all [ 2416.608310] LustreError: 75938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.623439] LustreError: 75938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2421.486924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.745210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2421.956548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.864726] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug +malloc all [ 2434.678984] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2441.929850] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2449.294536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.522967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.571309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2455.857633] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug +malloc all [ 2464.245327] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2477.541312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.547661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.549085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.556351] LustreError: Skipped 1 previous similar message [ 2477.570633] Lustre: Skipped 1 previous similar message [ 2482.657531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.674979] Lustre: Skipped 1 previous similar message [ 2482.906485] Lustre: server umount lustre-OST0000 complete [ 2487.777228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.780443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.786900] Lustre: Skipped 2 previous similar messages [ 2487.794512] Lustre: Skipped 1 previous similar message [ 2492.897743] Lustre: server umount lustre-MDT0000 complete [ 2492.899779] LustreError: 75939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.932985] LustreError: 75939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2497.129383] LustreError: 75918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828199 with bad export cookie 9996232927350124356 [ 2497.140906] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.485553] Lustre: server umount lustre-MDT0001 complete [ 2509.239411] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2513.054656] Key type lgssc unregistered [ 2513.513903] LNet: 79502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.528302] LNetError: 79502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.598658] LNet: Removed LNI 192.168.201.111@tcp [ 2515.737659] Key type .llcrypt unregistered [ 2515.741656] Key type ._llcrypt unregistered [ 2541.790753] Key type ._llcrypt registered [ 2541.795920] Key type .llcrypt registered [ 2541.937221] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2543.084364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.280072] alg: No test for adler32 (adler32-zlib) [ 2544.304504] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2544.536548] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2546.247173] Key type lgssc registered [ 2547.564246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.790224] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.801327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.400727] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2575.519313] LustreError: 80895:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.111@tcp: failed processing log, type 4: rc = -110 [ 2602.208726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.672346] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2618.209441] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2626.319902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.475205] Lustre: *** cfs_fail_loc=706, val=0*** [ 2627.050900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.157710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2631.952700] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2632.161303] LustreError: 80921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.169181] LustreError: 80921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2637.282982] LustreError: 81982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.331486] LustreError: 81982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2642.404701] LustreError: 81982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.459379] LustreError: 81982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2642.643251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.133777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.318917] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2660.594621] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2667.912230] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2673.126210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2673.143831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.172718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.144909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.163989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.173883] Lustre: Skipped 1 previous similar message [ 2677.797486] Lustre: server umount lustre-OST0000 complete [ 2684.385560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2684.389184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.404877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.420139] Lustre: Skipped 1 previous similar message [ 2684.445872] Lustre: Skipped 1 previous similar message [ 2688.922637] Lustre: server umount lustre-MDT0000 complete [ 2689.510652] LustreError: 81985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.529131] LustreError: 81985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2692.660739] LustreError: 80903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828394 with bad export cookie 13642111349468952786 [ 2692.666142] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2692.671897] LustreError: 80903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2693.092291] Lustre: server umount lustre-MDT0001 complete [ 2704.789480] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2707.796807] Key type lgssc unregistered [ 2708.084860] LNet: 84515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.091788] LNetError: 84515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.104264] LNet: Removed LNI 192.168.201.111@tcp [ 2708.954828] Key type .llcrypt unregistered [ 2708.956868] Key type ._llcrypt unregistered [ 2736.300448] Key type ._llcrypt registered [ 2736.302072] Key type .llcrypt registered [ 2736.397785] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2738.240381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2738.294159] alg: No test for adler32 (adler32-zlib) [ 2739.492535] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 2739.809705] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2741.511255] Key type lgssc registered [ 2742.648456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2751.046695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2751.068153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.103248] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2764.675036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.259769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2769.729369] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2770.403294] LustreError: 86619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.430620] LustreError: 86619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2775.523724] LustreError: 86622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.552641] LustreError: 86622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2777.727246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.045421] LustreError: 86619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.066312] LustreError: 86619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2778.074694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.213357] LustreError: 87366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.222111] LustreError: 87366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2785.000814] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2788.322707] LustreError: 86617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.346801] LustreError: 86617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2793.652987] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.673719] Lustre: MGS: Client 80d93aec-0d1c-4355-8924-99e2f68aa04d (at 0@lo) reconnecting [ 2793.688518] LustreError: 86633:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2794.021935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.091532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2799.419968] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2814.432937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.439449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.445479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.511260] Lustre: Skipped 1 previous similar message [ 2818.924776] Lustre: server umount lustre-OST0000 complete [ 2824.680139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.688065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2824.698174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.698183] Lustre: Skipped 1 previous similar message [ 2824.710952] Lustre: Skipped 1 previous similar message [ 2829.579925] Lustre: server umount lustre-MDT0000 complete [ 2829.803377] LustreError: 87369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.829692] LustreError: 87369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2835.206320] Lustre: server umount lustre-MDT0001 complete [ 2839.985960] Lustre: server umount MGS complete [ 2853.818733] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:15:54 (1786828554) [ 2885.642630] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 2895.968075] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2934.681745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.704495] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2949.226466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2949.614845] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2949.710523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2949.822168] Lustre: lustre-MDT0001: new disk, initializing [ 2949.928671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.960913] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2949.971578] Lustre: 93793:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2949.985449] LustreError: 93793:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2955.014823] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2966.165294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.439328] Lustre: lustre-OST0000: new disk, initializing [ 2966.447124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2966.452060] Lustre: 94559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2966.532158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.290912] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2974.753717] Lustre: 94933:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000060625ed0 x1873625419391232/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786828692 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2982.675667] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2990.938629] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2990.958358] Lustre: MGS: Client f2b7f94e-b7bd-4af3-bbd7-de238ae7ce07 (at 0@lo) reconnecting [ 2991.015624] Lustre: 95604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2991.061553] LustreError: 93779:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2991.070277] LustreError: 93779:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2991.114438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2991.226949] Lustre: lustre-MDT0000: new disk, initializing [ 2991.457952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.485063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2991.519658] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2992.545497] Lustre: 95628:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000c93edb7 x1873625419398016/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786828710 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2997.235618] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2997.791928] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3001.656593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3001.667514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3001.865734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3026.420042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.445064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.451055] Lustre: Skipped 2 previous similar messages [ 3029.542469] Lustre: server umount lustre-OST0000 complete [ 3036.647825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.649103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3036.668198] Lustre: Skipped 1 previous similar message [ 3036.671995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3040.541092] Lustre: server umount lustre-MDT0000 complete [ 3041.762611] LustreError: 95982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.786802] LustreError: 95982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3051.549327] Lustre: server umount lustre-MDT0001 complete [ 3056.909291] Lustre: server umount MGS complete [ 3072.804851] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:19:32 (1786828772) [ 3091.896226] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3094.588784] Key type lgssc unregistered [ 3094.830837] LNet: 98132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3094.838903] LNetError: 98132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3094.858224] LNet: Removed LNI 192.168.201.111@tcp [ 3095.770507] Key type .llcrypt unregistered [ 3095.774979] Key type ._llcrypt unregistered [ 3115.789758] Key type ._llcrypt registered [ 3115.791930] Key type .llcrypt registered [ 3115.912805] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3117.076899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3117.169393] alg: No test for adler32 (adler32-zlib) [ 3118.502339] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3118.809642] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3120.583197] Key type lgssc registered [ 3121.844593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3130.843284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3130.858022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.372959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.550800] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3142.633129] LustreError: 99352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.650519] LustreError: 99352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3147.759290] LustreError: 99353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.523977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.138924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3153.268691] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3163.220649] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3170.397969] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3178.032346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.429920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.562510] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3194.576872] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3201.971409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3215.329712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.341550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.349320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.376853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.382517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.391044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3217.398867] Lustre: Skipped 1 previous similar message [ 3219.436290] Lustre: server umount lustre-OST0000 complete [ 3226.081262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3226.088672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.098218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.138212] Lustre: server umount lustre-MDT0000 complete [ 3232.742120] LustreError: 99349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.753504] LustreError: 99349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3235.226835] LustreError: 99332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828937 with bad export cookie 9702615531144435890 [ 3235.231426] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.233373] LustreError: 99332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3235.428818] Lustre: server umount lustre-MDT0001 complete [ 3248.036190] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3250.629643] Key type lgssc unregistered [ 3251.057178] LNet: 102912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.070466] LNetError: 102912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.107105] LNet: Removed LNI 192.168.201.111@tcp [ 3252.400099] Key type .llcrypt unregistered [ 3252.403431] Key type ._llcrypt unregistered [ 3271.475901] Key type ._llcrypt registered [ 3271.480566] Key type .llcrypt registered [ 3271.606719] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3273.136333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3273.220678] alg: No test for adler32 (adler32-zlib) [ 3274.614826] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3275.250418] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3277.247167] Key type lgssc registered [ 3279.488071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3287.042400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3287.047938] LustreError: 103995:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3287.054078] Lustre: Skipped 1 previous similar message [ 3287.059279] LustreError: 103994:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3287.078244] LustreError: 103995:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 3287.092890] LustreError: 103995:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3287.097289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.103264] LustreError: 103995:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3288.824367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.149600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.277774] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3298.382716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.351813] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3310.624587] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing lsmod [ 3313.677746] LustreError: 105201:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3313.681171] LustreError: 105200:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3313.700550] LustreError: 105201:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3313.709523] LustreError: 105201:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3313.718604] LustreError: 105201:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3313.767519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.276272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3318.766864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3318.778608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.792433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3323.231202] Lustre: server umount lustre-MDT0000 complete [ 3323.375790] LustreError: 104033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.398338] LustreError: 104033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3327.351367] LustreError: 104008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829029 with bad export cookie 14348808435065757588 [ 3327.374539] LustreError: 104008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3327.376207] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.583301] LustreError: 104463:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000044f32ef4 x1873625981406720/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3327.878458] Lustre: server umount lustre-MDT0001 complete [ 3343.069972] Lustre: server umount lustre-OST0000 complete [ 3352.682793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.364839] LustreError: 106267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.449794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.619789] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3358.693352] LustreError: 106268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3363.815279] LustreError: 106267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.384360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.718385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.380817] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3379.842823] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3385.854487] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3393.345592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.776847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.910366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3401.419454] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3409.205541] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3422.696159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.704198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.712950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.722698] Lustre: Skipped 3 previous similar messages [ 3422.739261] Lustre: Skipped 4 previous similar messages [ 3427.811700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3427.828172] Lustre: Skipped 1 previous similar message [ 3428.619386] Lustre: server umount lustre-OST0000 complete [ 3432.929257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.932166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.953765] Lustre: Skipped 1 previous similar message [ 3432.971091] Lustre: Skipped 1 previous similar message [ 3443.170207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.185323] Lustre: Skipped 3 previous similar messages [ 3447.263368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.434855] Lustre: server umount lustre-MDT0000 complete [ 3448.290089] LustreError: 106268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.298768] LustreError: 106268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3452.069206] LustreError: 106247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829154 with bad export cookie 14348808435065758288 [ 3452.080247] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.084396] LustreError: 106247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3452.643800] Lustre: server umount lustre-MDT0001 complete [ 3464.699529] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3467.678299] Key type lgssc unregistered [ 3467.988616] LNet: 109828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3467.995666] LNetError: 109828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3468.013295] LNet: Removed LNI 192.168.201.111@tcp [ 3468.784117] Key type .llcrypt unregistered [ 3468.786485] Key type ._llcrypt unregistered [ 3495.611916] Key type ._llcrypt registered [ 3495.622453] Key type .llcrypt registered [ 3495.735749] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3496.789304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3496.817955] alg: No test for adler32 (adler32-zlib) [ 3497.848810] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3498.088899] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3499.775212] Key type lgssc registered [ 3500.801813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3508.774394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3508.786424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.312801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.169977] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3520.489076] LustreError: 111247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3520.507142] LustreError: 111247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3523.553033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3524.000793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3529.165668] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3538.095511] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3545.047511] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3552.895965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3553.353676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.447094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3560.583558] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3570.333876] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3575.600266] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3579.532073] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3585.507369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.523797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3585.540500] Lustre: Skipped 1 previous similar message [ 3590.625835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3590.627633] Lustre: Skipped 1 previous similar message [ 3595.754568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.762984] Lustre: Skipped 2 previous similar messages [ 3598.303151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3598.676700] Lustre: server umount lustre-OST0000 complete [ 3605.989694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3605.992060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.002551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3606.028865] Lustre: Skipped 1 previous similar message [ 3611.109656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.111725] Lustre: Skipped 1 previous similar message [ 3617.247324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3617.367086] Lustre: server umount lustre-MDT0000 complete [ 3620.460645] LustreError: 111226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829322 with bad export cookie 14585998061350156929 [ 3620.468404] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3620.474482] LustreError: 111226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3621.057916] Lustre: server umount lustre-MDT0001 complete [ 3633.103489] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3635.386282] Key type lgssc unregistered [ 3635.713774] LNet: 114892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3635.719192] LNetError: 114892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3635.735995] LNet: Removed LNI 192.168.201.111@tcp [ 3636.720982] Key type .llcrypt unregistered [ 3636.723305] Key type ._llcrypt unregistered [ 3652.379732] Key type ._llcrypt registered [ 3652.381140] Key type .llcrypt registered [ 3652.544989] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3653.993862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3654.034601] alg: No test for adler32 (adler32-zlib) [ 3655.124416] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3655.435041] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3657.207144] Key type lgssc registered [ 3658.328841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3665.199741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3665.216697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.535816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.240113] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3676.131116] LustreError: 116096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.154958] LustreError: 116096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3680.386021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.648421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.080823] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3693.812146] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3700.829966] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3708.873461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.236805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.307044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3715.544721] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3722.651829] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3736.038064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.055485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.059104] Lustre: Skipped 1 previous similar message [ 3736.071386] Lustre: Skipped 1 previous similar message [ 3741.158657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3741.165757] Lustre: Skipped 1 previous similar message [ 3741.512988] Lustre: server umount lustre-OST0000 complete [ 3746.277954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.281925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3746.290268] Lustre: Skipped 1 previous similar message [ 3746.300019] Lustre: Skipped 1 previous similar message [ 3751.400169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.244613] Lustre: server umount lustre-MDT0000 complete [ 3755.036743] LustreError: 116075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829457 with bad export cookie 6831322416371863037 [ 3755.049989] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.370264] Lustre: server umount lustre-MDT0001 complete [ 3765.092763] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3767.555543] Key type lgssc unregistered [ 3767.823701] LNet: 119654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3767.828764] LNetError: 119654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3768.871619] LNet: Removed LNI 192.168.201.111@tcp [ 3769.722470] Key type .llcrypt unregistered [ 3769.725352] Key type ._llcrypt unregistered [ 3791.617142] Key type ._llcrypt registered [ 3791.619354] Key type .llcrypt registered [ 3791.730866] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3793.320751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3793.385466] alg: No test for adler32 (adler32-zlib) [ 3794.480451] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3794.723025] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3796.376458] Key type lgssc registered [ 3797.594076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3805.277131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3805.289330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.664739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.629381] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3815.923269] LustreError: 121080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3815.944965] LustreError: 121080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3820.785482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.025345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3821.252428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3825.954494] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3834.745610] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3841.302566] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3849.045923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3849.393165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.440277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3856.252639] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3865.048899] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.031438] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3878.920189] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3878.924485] Lustre: Skipped 2 previous similar messages [ 3879.587795] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3879.648613] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3884.067277] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3884.080289] Lustre: Skipped 2 previous similar messages [ 3889.237495] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3889.247501] Lustre: Skipped 1 previous similar message [ 3895.960364] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.11@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3895.973395] Lustre: Skipped 2 previous similar messages [ 3897.687272] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.11@tcp 192.168.0.[2,10]@tcp [ 3902.974620] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.11/24@tcp in log lustre-MDT0000 [ 3902.981301] Lustre: Skipped 2 previous similar messages [ 3905.028894] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.11/24@tcp [ 3905.036347] Lustre: Skipped 1 previous similar message [ 3912.176095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3912.185194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.193993] LustreError: Skipped 2 previous similar messages [ 3912.217287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.200329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3913.217793] LustreError: Skipped 2 previous similar messages [ 3913.223565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.242929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.250957] Lustre: Skipped 1 previous similar message [ 3918.113260] Lustre: server umount lustre-OST0000 complete [ 3923.432925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.437352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.448062] Lustre: Skipped 1 previous similar message [ 3928.545378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.550350] Lustre: Skipped 2 previous similar messages [ 3936.223409] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3936.594185] Lustre: server umount lustre-MDT0000 complete [ 3938.785400] LustreError: 121081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.801651] LustreError: 121081:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3940.952599] LustreError: 121061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829643 with bad export cookie 13569636183896835618 [ 3940.961110] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.969459] LustreError: 121061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3941.455968] Lustre: server umount lustre-MDT0001 complete [ 3954.247417] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 3957.548260] Key type lgssc unregistered [ 3957.906580] LNet: 125883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3957.912200] LNetError: 125883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3958.967139] LNet: Removed LNI 192.168.201.111@tcp [ 3960.008247] Key type .llcrypt unregistered [ 3960.012364] Key type ._llcrypt unregistered [ 3988.012382] Key type ._llcrypt registered [ 3988.017231] Key type .llcrypt registered [ 3988.178163] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3989.393970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3989.433075] alg: No test for adler32 (adler32-zlib) [ 3990.793649] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 3990.989312] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 3992.680606] Key type lgssc registered [ 3994.526773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4003.830926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4003.856271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.418527] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4005.457736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4005.494128] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.11/24@tcp [ 4011.142377] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4015.585952] LustreError: 127445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.612487] LustreError: 127445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4020.723243] LustreError: 127446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.873366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.255335] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4022.297975] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.11/24@tcp [ 4022.359442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.436407] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4034.177567] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4040.064156] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4046.953748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4047.261640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4050.391937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4053.354795] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4060.409441] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4070.894701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4070.895861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.901398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4070.938547] Lustre: Skipped 1 previous similar message [ 4076.008116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.019312] Lustre: Skipped 2 previous similar messages [ 4079.017309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.020973] Lustre: Skipped 1 previous similar message [ 4084.191106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4084.195600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.201951] Lustre: Skipped 1 previous similar message [ 4084.318457] Lustre: server umount lustre-OST0000 complete [ 4088.800178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4088.806171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.814639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.432472] LustreError: 128485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.449692] LustreError: 128485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4095.655574] Lustre: server umount lustre-MDT0000 complete [ 4099.464358] LustreError: 127426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829801 with bad export cookie 13099739994806488059 [ 4099.483260] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.484479] LustreError: 127426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4099.801376] Lustre: server umount lustre-MDT0001 complete [ 4112.328665] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 4115.478446] Key type lgssc unregistered [ 4116.029097] LNet: 131037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.051227] LNetError: 131037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4117.101983] LNet: Removed LNI 192.168.201.111@tcp [ 4117.947469] Key type .llcrypt unregistered [ 4117.952818] Key type ._llcrypt unregistered [ 4143.432952] Key type ._llcrypt registered [ 4143.437378] Key type .llcrypt registered [ 4143.543667] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 4157.202868] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4158.466726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4158.493856] alg: No test for adler32 (adler32-zlib) [ 4159.797419] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 4160.052622] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 4161.840057] Key type lgssc registered [ 4163.647386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4201.102891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4201.142579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.462740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4202.481066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4202.588688] Lustre: lustre-MDT0000: new disk, initializing [ 4202.682375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.694500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4208.042803] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4222.240903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.319784] Lustre: 134443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4222.357510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4222.360382] Lustre: Skipped 1 previous similar message [ 4222.408534] Lustre: lustre-MDT0001: new disk, initializing [ 4222.510095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4222.574083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4222.588760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4226.463398] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4236.185246] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4242.555798] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4248.817308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.081794] Lustre: lustre-OST0000: new disk, initializing [ 4249.092124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4249.099994] Lustre: 136412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4249.211977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.695460] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4258.980479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4258.988539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4259.099599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4267.223743] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4271.632907] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4271.908654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4276.488634] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4276.751128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.777512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4279.781828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.784107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.802268] Lustre: Skipped 1 previous similar message [ 4284.830615] Lustre: server umount lustre-OST0000 complete [ 4288.995132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.004985] LustreError: Skipped 1 previous similar message [ 4289.014757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.042238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4289.046716] Lustre: Skipped 1 previous similar message [ 4290.033874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.941792] Lustre: server umount lustre-MDT0000 complete [ 4299.033593] LustreError: 134433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830001 with bad export cookie 10713961837095573870 [ 4299.036544] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4299.054236] LustreError: 134433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4299.432675] Lustre: server umount lustre-MDT0001 complete [ 4307.889361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.253975] LustreError: 138626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.267681] LustreError: 138626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4308.350799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.342430] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4313.575467] LustreError: 138627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.695448] LustreError: 138626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.730842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4321.187442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4325.381623] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4332.782812] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.536732] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4346.768886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4347.025854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4352.650507] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4360.473515] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4365.365811] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4365.595712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4370.400776] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4370.579995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4380.699980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4380.900850] Lustre: lustre-OST0001: new disk, initializing [ 4380.903866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4380.909592] Lustre: 141997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4381.010350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4387.669493] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4391.096213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4391.110924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4391.240460] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4396.821383] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4397.032197] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4401.569925] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4401.827616] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4408.523266] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965a8580f800.ost_server_uuid 50 [ 4410.486472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff965a8580f800.ost_server_uuid in IDLE state after 0 sec [ 4422.112539] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4422.114358] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.118329] LustreError: Skipped 1 previous similar message [ 4422.127135] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4422.131679] Lustre: Skipped 1 previous similar message [ 4422.157104] Lustre: Skipped 3 previous similar messages [ 4424.303992] Lustre: server umount lustre-OST0001 complete [ 4427.235707] LustreError: 141650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.267967] LustreError: 141650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4430.309730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4430.320797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.333904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.366231] LustreError: 140450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.388257] LustreError: 140450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4435.155758] Lustre: server umount lustre-OST0000 complete [ 4439.009703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4439.025858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.061111] Lustre: Skipped 1 previous similar message [ 4439.074786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.077995] Lustre: Skipped 2 previous similar messages [ 4447.713767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.733242] Lustre: Skipped 3 previous similar messages [ 4453.343248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4453.590545] Lustre: server umount lustre-MDT0000 complete [ 4457.954650] LustreError: 138627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.996313] LustreError: 138627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4458.080802] LustreError: 138607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830160 with bad export cookie 10713961837095574752 [ 4458.082135] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.096933] LustreError: 138607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4458.680864] Lustre: server umount lustre-MDT0001 complete [ 4479.775274] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 4482.653749] Key type lgssc unregistered [ 4483.088687] LNet: 145006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4483.095399] LNetError: 145006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4484.140215] LNet: Removed LNI 192.168.201.111@tcp [ 4485.287470] Key type .llcrypt unregistered [ 4485.292265] Key type ._llcrypt unregistered [ 4529.985208] Key type ._llcrypt registered [ 4529.987683] Key type .llcrypt registered [ 4530.076584] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 4548.570286] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4550.168048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4550.241651] alg: No test for adler32 (adler32-zlib) [ 4551.312216] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 4551.556206] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 4553.207154] Key type lgssc registered [ 4554.447834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4593.995322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4594.041719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.289620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4595.327860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4595.445793] Lustre: lustre-MDT0000: new disk, initializing [ 4595.560977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.577991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4600.576737] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4614.078023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.324568] Lustre: 149172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4614.458465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4614.465537] Lustre: Skipped 1 previous similar message [ 4614.592382] Lustre: lustre-MDT0001: new disk, initializing [ 4614.749870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4614.794147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4614.808258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4619.506230] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4632.829193] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4639.954877] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4646.681388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4647.079672] Lustre: lustre-OST0000: new disk, initializing [ 4647.082807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4647.091590] Lustre: 151140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4647.201904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.132433] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4653.667058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4653.679500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4653.789206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4666.129823] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4672.576727] Lustre: Failing over lustre-OST0000 [ 4672.708433] Lustre: server umount lustre-OST0000 complete [ 4689.887090] Lustre: 147397:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830376/real 1786830376] req@000000007e52d66c x1873627319406208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786830392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4689.913607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.906696] Lustre: 147395:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830376/real 1786830376] req@000000000355cdd3 x1873627319406336/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786830392 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4690.933341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.050830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.346424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.376508] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4695.391705] Lustre: 147396:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830381/real 1786830381] req@000000007740f4ea x1873627319407360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786830397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4695.411445] Lustre: 147396:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4696.298457] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4696.893951] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4700.768569] Lustre: Failing over lustre-MDT0000 [ 4701.117645] Lustre: server umount lustre-MDT0000 complete [ 4701.433823] LustreError: 152061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.448283] LustreError: 152061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4701.664679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.679180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.775110] Lustre: 147396:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830386/real 1786830386] req@0000000049cc729e x1873627319408512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786830402 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4701.793259] Lustre: 147396:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4705.761020] LustreError: 149183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.763849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.791251] LustreError: 149183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4705.823177] Lustre: Skipped 1 previous similar message [ 4706.783200] Lustre: 147398:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830392/real 1786830392] req@000000002624b725 x1873627319409664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786830408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4710.880238] LustreError: 150341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.894980] LustreError: 150341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4716.004660] LustreError: 152061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.019137] LustreError: 152061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4720.617876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.701661] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.942125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4720.996570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.035618] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4722.900242] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4725.874045] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4726.270227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4726.280861] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4726.325914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4736.483744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4736.486990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.500461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4736.511676] Lustre: Skipped 1 previous similar message [ 4736.531297] Lustre: Skipped 1 previous similar message [ 4740.058749] Lustre: server umount lustre-OST0000 complete [ 4746.721497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.725252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.731217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4746.743607] Lustre: Skipped 1 previous similar message [ 4746.762359] LustreError: Skipped 1 previous similar message [ 4750.590148] Lustre: server umount lustre-MDT0000 complete [ 4751.845058] LustreError: 149184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.853499] LustreError: 149184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4754.105515] LustreError: 154042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830456 with bad export cookie 18339483962666435072 [ 4754.117447] LustreError: 154042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4754.120125] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.772943] Lustre: server umount lustre-MDT0001 complete [ 4766.279445] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 4769.322465] Key type lgssc unregistered [ 4769.536757] LNet: 154797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4769.541157] LNetError: 154797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4769.558190] LNet: Removed LNI 192.168.201.111@tcp [ 4770.191606] Key type .llcrypt unregistered [ 4770.193161] Key type ._llcrypt unregistered [ 4803.189742] Key type ._llcrypt registered [ 4803.193925] Key type .llcrypt registered [ 4803.354548] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4804.353496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4804.387115] alg: No test for adler32 (adler32-zlib) [ 4805.373818] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 4805.567121] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 4807.311187] Key type lgssc registered [ 4808.332784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4814.839927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4814.853897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.503650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4820.963912] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4826.086214] LustreError: 156323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4826.106454] LustreError: 156323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4829.225333] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4831.208212] LustreError: 156324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.237435] LustreError: 156352:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4833.242639] LustreError: 156352:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4833.248425] LustreError: 156352:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4833.515926] Lustre: server umount lustre-MDT0000 complete [ 4850.941938] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 4861.882309] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4900.886362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.198490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4901.220179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4901.321961] Lustre: lustre-MDT0000: new disk, initializing [ 4901.443849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4901.461322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4907.247426] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4919.071982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.242683] Lustre: 159655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4919.272390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4919.274430] Lustre: Skipped 1 previous similar message [ 4919.358627] Lustre: lustre-MDT0001: new disk, initializing [ 4919.468765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4919.495387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4919.504154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4924.237076] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4934.106866] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4940.000925] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4949.035621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4949.357440] Lustre: lustre-OST0000: new disk, initializing [ 4949.372317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4949.379624] Lustre: 161624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4949.479338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.891205] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4959.335873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4959.351787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4959.501215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4966.260457] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4971.137587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4973.229911] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:51:13 (1786830673) [ 5225.959976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.966501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5225.981565] Lustre: Skipped 2 previous similar messages [ 5225.997399] Lustre: Skipped 2 previous similar messages [ 5230.998756] Lustre: server umount lustre-MDT0000 complete [ 5231.077202] LustreError: 159662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5231.102748] LustreError: 159662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5234.409921] LustreError: 160449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830936 with bad export cookie 650430305468112495 [ 5234.410861] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5234.415065] LustreError: 160449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5236.192125] LustreError: 161217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.195379] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.213427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5239.209376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5240.864823] Lustre: server umount lustre-MDT0001 complete [ 5251.473776] Lustre: server umount lustre-OST0000 complete [ 5261.878480] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 5274.033322] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5316.011963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.288600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5316.340992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5316.468461] Lustre: lustre-MDT0000: new disk, initializing [ 5316.594975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.613685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5321.499663] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5334.569461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.808392] Lustre: 166291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5334.897584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5334.903393] Lustre: Skipped 1 previous similar message [ 5334.997954] Lustre: lustre-MDT0001: new disk, initializing [ 5335.138562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5335.185747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5335.201524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5339.443076] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5349.905708] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5356.140842] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5365.501296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5365.938526] Lustre: lustre-OST0000: new disk, initializing [ 5365.944714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5365.951575] Lustre: 168261:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5366.107421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.705281] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5375.560907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5375.578982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5375.824246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5384.559300] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5390.207424] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5390.581881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5398.340127] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5398.551141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5401.574177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5401.574915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5401.597967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5406.690963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5406.704067] Lustre: Skipped 2 previous similar messages [ 5415.391132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5415.619580] Lustre: server umount lustre-OST0000 complete [ 5422.053529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5422.068500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5422.069556] Lustre: Skipped 2 previous similar messages [ 5422.094932] Lustre: Skipped 3 previous similar messages [ 5434.335306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5436.616660] Lustre: server umount lustre-MDT0000 complete [ 5437.423778] LustreError: 166301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5437.463939] LustreError: 166301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5441.245440] LustreError: 167085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831143 with bad export cookie 650430305468271325 [ 5441.248895] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.260520] LustreError: 167085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5441.669624] Lustre: server umount lustre-MDT0001 complete [ 5450.458661] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:59:11 (1786831151) [ 5467.427619] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 5477.543106] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5524.353778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.601175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5524.630923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5524.708795] Lustre: lustre-MDT0000: new disk, initializing [ 5524.808305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5524.833198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5530.261042] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5541.655559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.836695] Lustre: 173020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5542.256344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5547.794784] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5558.462163] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5565.022646] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5571.846106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5572.241647] Lustre: lustre-OST0000: new disk, initializing [ 5572.244349] Lustre: Skipped 1 previous similar message [ 5572.246941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5572.250269] Lustre: Skipped 2 previous similar messages [ 5572.254975] Lustre: 175010:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5578.944525] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5580.369412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5580.385976] Lustre: Skipped 1 previous similar message [ 5580.394072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5580.467458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5589.207314] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5601.262772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.272807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.281218] LustreError: Skipped 1 previous similar message [ 5601.292708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.292716] Lustre: Skipped 4 previous similar messages [ 5602.801204] Lustre: server umount lustre-OST0000 complete [ 5608.928524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5611.496626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5611.508267] Lustre: Skipped 2 previous similar messages [ 5621.215195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5621.364264] Lustre: server umount lustre-MDT0000 complete [ 5621.729448] LustreError: 174393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.746722] LustreError: 174393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5625.750060] LustreError: 173013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831327 with bad export cookie 650430305468272207 [ 5625.762606] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5625.771750] LustreError: 173013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5626.339554] Lustre: server umount lustre-MDT0001 complete [ 5635.143747] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:02:15 (1786831335) [ 5653.938744] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 5666.608315] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5703.855235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5704.117727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5704.154529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5704.253912] Lustre: lustre-MDT0000: new disk, initializing [ 5704.333363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5704.340928] Lustre: Skipped 2 previous similar messages [ 5704.372312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5708.711060] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5718.948196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.116789] Lustre: 179335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5719.579522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5724.264266] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5734.001673] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5740.120340] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.628738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.092550] Lustre: 181319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5754.967172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5755.172932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5755.236397] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5765.526527] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5778.801392] Lustre: server umount lustre-OST0000 complete [ 5786.084862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5786.085964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5786.086838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.086843] Lustre: Skipped 6 previous similar messages [ 5786.094104] Lustre: Skipped 1 previous similar message [ 5797.343341] Lustre: 155637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831483/real 1786831483] req@00000000b18c569f x1873627586526720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786831499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5797.380076] Lustre: 155637:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5798.367195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5798.583414] Lustre: server umount lustre-MDT0000 complete [ 5802.369118] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831504 with bad export cookie 650430305468273292 [ 5802.373907] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5802.379569] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5802.633418] Lustre: server umount lustre-MDT0001 complete [ 5814.055558] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 5817.017912] Key type lgssc unregistered [ 5817.376697] LNet: 183179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.379397] LNetError: 183179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5817.392785] LNet: Removed LNI 192.168.201.111@tcp [ 5818.491445] Key type .llcrypt unregistered [ 5818.495172] Key type ._llcrypt unregistered [ 5846.293311] Key type ._llcrypt registered [ 5846.304944] Key type .llcrypt registered [ 5846.474789] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 5848.438980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5848.698907] alg: No test for adler32 (adler32-zlib) [ 5850.279395] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 5850.658136] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 5852.391281] Key type lgssc registered [ 5854.218068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5864.731813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.767196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.079366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.410880] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5876.715646] LustreError: 184600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.744188] LustreError: 184600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5880.151854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5880.619994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5885.638761] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5894.402289] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5899.996866] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5907.493114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5907.947863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.033414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5914.241369] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 5923.176579] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5945.830666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.831187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5945.840939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.854486] Lustre: Skipped 1 previous similar message [ 5946.219942] Lustre: server umount lustre-OST0000 complete [ 5950.951718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.957166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5950.971401] Lustre: Skipped 1 previous similar message [ 5950.991352] Lustre: Skipped 2 previous similar messages [ 5956.065543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5956.071713] Lustre: Skipped 1 previous similar message [ 5961.186848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5961.191617] Lustre: Skipped 1 previous similar message [ 5964.255214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5964.517515] Lustre: server umount lustre-MDT0000 complete [ 5966.326432] LustreError: 184599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.865548] LustreError: 184581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831672 with bad export cookie 2251908583334271053 [ 5969.875076] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5969.878421] LustreError: 184581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5970.482394] Lustre: server umount lustre-MDT0001 complete [ 5985.298707] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 5988.402579] Key type lgssc unregistered [ 5988.675191] LNet: 188162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5988.684761] LNetError: 188162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5988.703560] LNet: Removed LNI 192.168.201.111@tcp [ 5990.079744] Key type .llcrypt unregistered [ 5990.090419] Key type ._llcrypt unregistered [ 6019.209605] Key type ._llcrypt registered [ 6019.216276] Key type .llcrypt registered [ 6019.369264] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 6020.900236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6020.970860] alg: No test for adler32 (adler32-zlib) [ 6022.095546] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 6022.395397] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 6024.127215] Key type lgssc registered [ 6025.227097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6033.418521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6033.439425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.922432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.172526] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6045.161922] LustreError: 189578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.173132] LustreError: 189578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6050.283075] LustreError: 189579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.068273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.566964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6056.258924] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6065.244791] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6072.911793] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6082.290659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6082.830957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6090.348648] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6100.224840] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6105.467953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6111.715173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6111.723359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.739888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6115.819130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6115.827663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.827846] Lustre: Skipped 1 previous similar message [ 6115.827896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6117.082385] Lustre: lustre-OST0000: Not available for connect from 192.168.201.11@tcp (stopping) [ 6117.099299] Lustre: Skipped 1 previous similar message [ 6117.893139] Lustre: server umount lustre-OST0000 complete [ 6126.771951] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff965aa00c6800.ost_server_uuid 50 [ 6128.847938] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6143.984503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6143.992990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.005392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.535811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6150.077371] Lustre: server umount lustre-MDT0000 complete [ 6151.654575] LustreError: 191326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6151.684900] LustreError: 191326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6155.061153] LustreError: 189559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831857 with bad export cookie 5506573509930984048 [ 6155.062437] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.080724] LustreError: 189559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6155.596779] Lustre: server umount lustre-MDT0001 complete [ 6166.809993] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:11:06 (1786831866) [ 6173.956077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.511439] LustreError: 193498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.583579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.814176] LustreError: 193499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.214289] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6189.024471] LustreError: 193499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.055069] LustreError: 193499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6190.789084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.427032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6196.925514] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6205.768562] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.894622] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6222.489589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.004459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.399656] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6241.989715] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6244.551631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6251.130328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6251.359574] Lustre: lustre-OST0001: new disk, initializing [ 6251.364239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6251.373898] Lustre: 196341:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6251.508269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.876900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6259.898134] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6260.011146] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6260.175340] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6272.426111] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6277.613835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6277.641099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.661035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6277.666470] Lustre: Skipped 2 previous similar messages [ 6282.963462] Lustre: server umount lustre-OST0000 complete [ 6284.004278] LustreError: 195323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.280880] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6291.624286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6297.006521] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6297.260758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6311.397381] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6311.405139] LustreError: Skipped 1 previous similar message [ 6311.412348] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.429232] Lustre: Skipped 1 previous similar message [ 6311.439794] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6311.447355] Lustre: Skipped 4 previous similar messages [ 6316.522057] LustreError: 195323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.541266] LustreError: 195323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6317.459151] Lustre: server umount lustre-OST0001 complete [ 6324.706731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6324.717917] LustreError: Skipped 1 previous similar message [ 6328.280841] Lustre: server umount lustre-MDT0000 complete [ 6331.927234] LustreError: 196340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832034 with bad export cookie 5506573509930985119 [ 6331.937953] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.941870] LustreError: 196340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6332.501796] Lustre: server umount lustre-MDT0001 complete [ 6359.734858] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:14:19 (1786832059) [ 6369.861236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.474746] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6370.529962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6370.780390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.396684] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6386.718047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.917360] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6386.977836] Lustre: 199787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6391.660259] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6400.424078] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6407.285789] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6414.982378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.193564] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6422.108740] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6423.746344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6430.935373] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6438.479962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6438.578583] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6438.875596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6438.878027] Lustre: Skipped 2 previous similar messages [ 6444.768990] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6452.824728] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6456.811121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.829641] Lustre: Skipped 3 previous similar messages [ 6456.835116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6456.837576] Lustre: Skipped 5 previous similar messages [ 6461.905723] Lustre: server umount lustre-OST0000 complete [ 6461.940212] LustreError: 202430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.970066] LustreError: 202430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6481.888884] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6484.630089] Lustre: server umount lustre-OST0001 complete [ 6489.583716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.600760] Lustre: Skipped 3 previous similar messages [ 6489.605901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.614114] Lustre: Skipped 4 previous similar messages [ 6503.391208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6503.878771] Lustre: server umount lustre-MDT0000 complete [ 6509.524342] LustreError: 200532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832211 with bad export cookie 5506573509930986526 [ 6509.530594] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6509.535237] LustreError: 200532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6509.876194] Lustre: server umount lustre-MDT0001 complete [ 6536.931915] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:17:17 (1786832237) [ 6553.134124] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 6563.897867] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 6609.313467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.628699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6609.668541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6609.785081] Lustre: lustre-MDT0000: new disk, initializing [ 6609.881101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.909306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6615.206155] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6630.218236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.380549] Lustre: 207883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6630.549566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6630.558381] Lustre: Skipped 1 previous similar message [ 6630.681616] Lustre: lustre-MDT0001: new disk, initializing [ 6630.861037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6630.876994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6637.527903] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6646.809845] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6653.766547] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6663.009063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.281302] Lustre: lustre-OST0000: new disk, initializing [ 6663.285142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6663.293253] Lustre: 209854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6670.174193] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6671.909392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6671.914603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6671.989561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6679.723530] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.399692] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6685.572215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6690.739752] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6690.932746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6697.455781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.456029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6697.483564] Lustre: Skipped 1 previous similar message [ 6697.484488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6697.503312] LustreError: Skipped 2 previous similar messages [ 6697.531669] Lustre: Skipped 7 previous similar messages [ 6699.261360] Lustre: server umount lustre-OST0000 complete [ 6710.458594] Lustre: server umount lustre-MDT0000 complete [ 6712.819938] LustreError: 207890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6712.863235] LustreError: 207890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6714.848822] LustreError: 207874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832416 with bad export cookie 5506573509930988087 [ 6714.857201] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6714.866611] LustreError: 207874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6715.303812] Lustre: server umount lustre-MDT0001 complete [ 6723.631025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.845970] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6737.394375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.057977] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6752.288985] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.494953] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6766.539485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6774.409299] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6782.933922] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.408691] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6788.686432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6793.100203] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6793.361906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6797.792526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6797.816334] LustreError: Skipped 1 previous similar message [ 6801.562966] Lustre: server umount lustre-OST0000 complete [ 6808.712939] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6814.146988] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6844.418730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.432100] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6860.233793] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6863.147040] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6863.469904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6863.487528] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6863.488175] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6868.974485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.984633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6868.988118] Lustre: Skipped 6 previous similar messages [ 6869.007434] Lustre: Skipped 5 previous similar messages [ 6874.617678] Lustre: server umount lustre-OST0000 complete [ 6885.991935] Lustre: server umount lustre-MDT0000 complete [ 6890.976048] LustreError: 212051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832593 with bad export cookie 5506573509930988969 [ 6890.984123] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6891.004598] LustreError: 212051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6891.500579] Lustre: server umount lustre-MDT0001 complete [ 6900.996688] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:23:21 (1786832601) [ 6908.586866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.244919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6909.250190] Lustre: Skipped 6 previous similar messages [ 6915.005250] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6924.889692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6929.775468] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6938.941878] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6945.337404] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6955.514366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.461978] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6973.819884] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6978.851049] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6980.229139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6985.589561] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6985.885037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6995.746924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.043795] Lustre: lustre-OST0001: new disk, initializing [ 6996.047039] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6996.057564] Lustre: 221419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7005.828054] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7005.941371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7005.972581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7006.223718] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7019.055880] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.304357] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7024.475744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7029.263914] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7029.433473] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7031.776520] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.787750] LustreError: Skipped 3 previous similar messages [ 7037.421405] Lustre: server umount lustre-OST0001 complete [ 7042.042625] LustreError: 219862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.060457] LustreError: 219862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 7044.121669] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7044.376605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7049.248313] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7049.444596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7078.671672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.018190] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7084.046163] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7084.046618] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7084.081457] Lustre: Skipped 1 previous similar message [ 7087.007571] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7094.382268] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.219837] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7104.633372] Lustre: server umount lustre-OST0001 complete [ 7127.632699] Lustre: server umount lustre-OST0000 complete [ 7133.159766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7133.166880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7133.177024] Lustre: Skipped 9 previous similar messages [ 7133.192467] Lustre: Skipped 16 previous similar messages [ 7137.291567] Lustre: server umount lustre-MDT0000 complete [ 7141.191178] LustreError: 218020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832843 with bad export cookie 5506573509930989928 [ 7141.204624] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.686835] Lustre: server umount lustre-MDT0001 complete [ 7174.182645] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:27:54 (1786832874) [ 7181.213854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.549121] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7181.595068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7187.440316] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7197.676994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.849199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7197.891799] Lustre: 226666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7203.059502] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7211.605922] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7217.583167] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7224.759943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7224.927739] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7233.095962] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7241.825102] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7254.032081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.148514] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7262.718370] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7272.879462] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7275.840716] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7283.086783] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7283.611885] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7289.225547] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7289.444619] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7296.759986] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff965a98ac0800.ost_server_uuid 50 [ 7299.047659] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff965a98ac0800.ost_server_uuid in IDLE state after 0 sec [ 7300.356644] Lustre: Permanently deactivating lustre-OST0001 [ 7300.393319] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7304.696389] Lustre: Permanently reactivating lustre-OST0001 [ 7304.720463] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7304.725198] Lustre: Skipped 2 previous similar messages [ 7310.315558] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7310.318118] LustreError: Skipped 3 previous similar messages [ 7315.873849] Lustre: server umount lustre-OST0001 complete [ 7327.023648] Lustre: server umount lustre-OST0000 complete [ 7337.601529] Lustre: server umount lustre-MDT0000 complete [ 7342.149577] LustreError: 226657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833044 with bad export cookie 5506573509930991223 [ 7342.151892] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7342.159424] LustreError: 226657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7342.948297] Lustre: server umount lustre-MDT0001 complete [ 7375.942404] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:31:16 (1786833076) [ 7385.719873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.089755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7386.155138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7386.169812] Lustre: Skipped 2 previous similar messages [ 7392.438372] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7403.238058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.347164] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7403.401866] Lustre: 233372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7409.610284] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7418.566552] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7426.386730] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7435.819153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7436.020203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7436.049994] Lustre: Permanently deactivating lustre-OST0000 [ 7436.082729] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7436.346483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.357821] Lustre: Skipped 10 previous similar messages [ 7442.989358] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7444.734500] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7454.915651] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7463.193792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7463.254166] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7470.882465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7471.117957] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7479.975734] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7482.115812] Lustre: Permanently reactivating lustre-OST0000 [ 7482.143884] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7482.151838] Lustre: Skipped 2 previous similar messages [ 7491.294513] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 48 seconds. I think it's dead, and I am evicting it. exp 000000007c01a750, cur 1786833193 deadline 1786833175 last 1786833145 [ 7491.777898] LustreError: 235220:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fc4fb296 x1873628862362496/t0(0) o104->lustre-OST0001@192.168.201.11@tcp:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 7491.837353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7500.521149] Lustre: server umount lustre-OST0001 complete [ 7511.937660] Lustre: server umount lustre-OST0000 complete [ 7522.055519] Lustre: server umount lustre-MDT0000 complete [ 7527.190224] LustreError: 236967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833229 with bad export cookie 5506573509930992889 [ 7527.194915] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.198184] LustreError: 236967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7527.583852] Lustre: server umount lustre-MDT0001 complete [ 7543.932331] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 7547.490906] Key type lgssc unregistered [ 7547.795170] LNet: 238238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.800712] LNetError: 238238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.815496] LNet: Removed LNI 192.168.201.111@tcp [ 7548.682361] Key type .llcrypt unregistered [ 7548.686050] Key type ._llcrypt unregistered [ 7577.137428] Key type ._llcrypt registered [ 7577.139821] Key type .llcrypt registered [ 7577.302887] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7578.585886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7578.733365] alg: No test for adler32 (adler32-zlib) [ 7579.894310] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 7580.202880] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 7581.975472] Key type lgssc registered [ 7583.665772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7595.235631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7595.282991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.873587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7601.553139] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7606.243672] LustreError: 239707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7606.267811] LustreError: 239707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7610.657129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.815763] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7610.849938] Lustre: 239696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7610.861735] Lustre: Permanently deactivating lustre-MDT0001 [ 7610.879706] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7611.245444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7611.292525] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7611.295462] LustreError: 239738:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7611.319501] LustreError: 239738:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7611.358875] LustreError: 239738:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7617.026404] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7625.988378] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7632.962861] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7641.325233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.763837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.688167] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7656.140395] LustreError: 241550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.267893] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.541053] LustreError: 241548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.662448] LustreError: 241549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.690157] LustreError: 241549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7666.698920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7668.007477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7668.198745] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7673.322545] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7673.325742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7673.546986] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7680.954404] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7685.572659] Lustre: Permanently reactivating lustre-MDT0001 [ 7685.599582] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7685.608825] Lustre: Skipped 2 previous similar messages [ 7687.499272] Lustre: lustre-MDT0001: Client fa076cb8-6cd3-42da-aed3-08c93acd4a0d (at 192.168.201.11@tcp) reconnecting [ 7695.316053] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7697.620206] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7705.002530] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7705.421043] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7711.412594] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7711.847037] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7715.200503] Lustre: Permanently deactivating lustre-MDT0001 [ 7715.211688] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7715.214938] Lustre: Skipped 2 previous similar messages [ 7724.870395] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7730.968848] Lustre: server umount lustre-MDT0000 complete [ 7734.243289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7734.252088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.271705] LustreError: 241862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.756458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7734.773109] Lustre: Skipped 2 previous similar messages [ 7735.202454] LustreError: 239689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833437 with bad export cookie 11689577801584207254 [ 7735.215958] LustreError: 239689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7735.221476] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7735.742761] Lustre: server umount lustre-MDT0001 complete [ 7740.179287] Lustre: server umount lustre-OST0000 complete [ 7756.578363] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:37:36 (1786833456) [ 7770.012107] Lustre: server umount lustre-OST0001 complete [ 7779.005842] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 7796.046821] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7841.691468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7842.241684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7842.264901] Lustre: Skipped 2 previous similar messages [ 7842.348638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7842.501933] Lustre: lustre-MDT0000: new disk, initializing [ 7842.656223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.722251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7848.668586] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7860.903109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.090911] Lustre: 247991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7861.173973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7861.176772] Lustre: Skipped 1 previous similar message [ 7861.296899] Lustre: lustre-MDT0001: new disk, initializing [ 7861.438682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7861.492743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7861.516700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7866.111922] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7876.705781] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7883.164639] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7891.026531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7891.494765] Lustre: lustre-OST0000: new disk, initializing [ 7891.498652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7891.505550] Lustre: 249962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7891.631766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7899.674237] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7899.780958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7899.784029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7899.989541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7911.773560] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7914.594187] LustreError: 249290:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7924.608060] LustreError: 249290:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7924.663363] LustreError: 247997:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7925.063912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7925.297228] Lustre: lustre-OST0001: new disk, initializing [ 7925.301103] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7925.308205] Lustre: 251077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7925.448658] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7932.497507] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7934.631897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7934.648794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7934.699026] LustreError: 247997:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7934.770303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7945.138644] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7946.778951] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7950.307461] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7950.312793] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7950.345967] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7955.226881] Lustre: server umount lustre-OST0001 complete [ 7955.461794] LustreError: 249943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.484839] LustreError: 249943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7975.923845] LustreError: 249946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7975.934075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.936992] LustreError: 249946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7975.965480] Lustre: Skipped 1 previous similar message [ 7975.974084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7975.982813] Lustre: Skipped 2 previous similar messages [ 7978.880321] Lustre: server umount lustre-OST0000 complete [ 7984.608796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7984.626325] LustreError: Skipped 1 previous similar message [ 7984.640095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7984.651919] Lustre: Skipped 1 previous similar message [ 7984.661565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7989.204332] Lustre: server umount lustre-MDT0000 complete [ 7993.601336] LustreError: 247985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833695 with bad export cookie 11689577801584209851 [ 7993.603917] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7993.631265] LustreError: 247985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8000.046402] Lustre: server umount lustre-MDT0001 complete [ 8014.291276] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8017.903803] Key type lgssc unregistered [ 8018.254911] LNet: 253116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.262950] LNetError: 253116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8019.317806] LNet: Removed LNI 192.168.201.111@tcp [ 8020.513950] Key type .llcrypt unregistered [ 8020.516282] Key type ._llcrypt unregistered [ 8066.945676] Key type ._llcrypt registered [ 8066.948285] Key type .llcrypt registered [ 8067.142184] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8069.776835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8069.821764] alg: No test for adler32 (adler32-zlib) [ 8071.163195] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 8071.577906] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8073.399188] Key type lgssc registered [ 8074.663717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8088.446234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8088.475476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.868551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8089.932872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8090.187097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.095305] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8108.954577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.118247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8109.151677] Lustre: 255327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8109.578138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8116.783692] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8124.997578] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8131.956589] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8139.844701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.098072] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8140.367154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.643552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8149.814504] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8160.641125] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8217.568493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8217.570252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.575531] LustreError: Skipped 1 previous similar message [ 8217.578015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8217.610962] Lustre: Skipped 1 previous similar message [ 8222.365591] Lustre: server umount lustre-OST0000 complete [ 8226.532303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8237.857257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8238.158856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8243.131331] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8249.594808] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8257.517735] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8257.534350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8257.549792] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8257.567759] Lustre: Skipped 1 previous similar message [ 8293.345835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8293.348431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8293.363275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8293.366417] Lustre: Skipped 1 previous similar message [ 8295.406371] Lustre: server umount lustre-OST0000 complete [ 8303.596864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8303.614834] Lustre: Skipped 2 previous similar messages [ 8303.620182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8303.630743] Lustre: Skipped 2 previous similar messages [ 8308.706821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8308.715366] Lustre: Skipped 1 previous similar message [ 8313.824846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8313.833167] Lustre: Skipped 1 previous similar message [ 8314.335149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8314.570140] Lustre: server umount lustre-MDT0000 complete [ 8318.398353] LustreError: 255317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834020 with bad export cookie 7108547511528061801 [ 8318.411990] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8318.681133] Lustre: server umount lustre-MDT0001 complete [ 8330.708875] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8334.237659] Key type lgssc unregistered [ 8334.431449] LNet: 260760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8334.444759] LNetError: 260760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8334.466419] LNet: Removed LNI 192.168.201.111@tcp [ 8335.477932] Key type .llcrypt unregistered [ 8335.482093] Key type ._llcrypt unregistered [ 8363.613186] Key type ._llcrypt registered [ 8363.614898] Key type .llcrypt registered [ 8363.741150] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8365.637013] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8365.673220] alg: No test for adler32 (adler32-zlib) [ 8366.676701] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 8366.828064] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8368.512567] Key type lgssc registered [ 8369.426495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8377.456780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8377.479818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.370917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8384.705309] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8389.614458] LustreError: 262180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.644369] LustreError: 262180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8394.730154] LustreError: 262181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.885205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.177833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8401.056793] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8410.405771] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8417.512886] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8425.497113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8425.952242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.627411] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8441.058097] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8450.567745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8481.249576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8481.252796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.255609] LustreError: Skipped 1 previous similar message [ 8481.265842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8481.270715] Lustre: Skipped 1 previous similar message [ 8481.280768] Lustre: Skipped 1 previous similar message [ 8483.274622] Lustre: server umount lustre-OST0000 complete [ 8491.489217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8491.510367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8491.514796] Lustre: Skipped 1 previous similar message [ 8494.070577] Lustre: server umount lustre-MDT0000 complete [ 8496.611217] LustreError: 262175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8496.626034] LustreError: 262175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8498.103236] LustreError: 262161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834200 with bad export cookie 6254461353989875367 [ 8498.117333] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8498.977240] Lustre: server umount lustre-MDT0001 complete [ 8510.033815] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8512.856960] Key type lgssc unregistered [ 8513.108761] LNet: 266512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8513.112859] LNetError: 266512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8513.125237] LNet: Removed LNI 192.168.201.111@tcp [ 8513.800114] Key type .llcrypt unregistered [ 8513.803063] Key type ._llcrypt unregistered [ 8532.979961] Key type ._llcrypt registered [ 8532.981562] Key type .llcrypt registered [ 8533.099279] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8534.629585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8534.682362] alg: No test for adler32 (adler32-zlib) [ 8535.881868] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 8536.172443] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8537.871194] Key type lgssc registered [ 8539.167136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8546.948989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8546.960447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8548.252174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8552.633687] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8558.569400] LustreError: 267734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8558.601425] LustreError: 267734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8561.663699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.198463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8567.376150] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8576.152398] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8583.096368] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8589.652674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8589.985734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.098927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8596.754784] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8605.055560] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8617.971955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8617.974636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8617.985209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8618.017480] Lustre: Skipped 1 previous similar message [ 8622.016872] Lustre: server umount lustre-OST0000 complete [ 8628.201839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.212058] Lustre: Skipped 1 previous similar message [ 8628.218061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8628.226478] Lustre: Skipped 1 previous similar message [ 8631.922172] Lustre: server umount lustre-MDT0000 complete [ 8633.312370] LustreError: 268462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8633.333382] LustreError: 268462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8635.791973] LustreError: 267715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834337 with bad export cookie 7953630172751682710 [ 8635.797673] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8635.808785] LustreError: 267715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8636.180557] Lustre: server umount lustre-MDT0001 complete [ 8647.183814] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8649.560577] Key type lgssc unregistered [ 8649.755720] LNet: 271412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8649.760821] LNetError: 271412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8650.790702] LNet: Removed LNI 192.168.201.111@tcp [ 8651.569969] Key type .llcrypt unregistered [ 8651.573313] Key type ._llcrypt unregistered [ 8675.667945] Key type ._llcrypt registered [ 8675.669731] Key type .llcrypt registered [ 8675.780297] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8676.828701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8677.044794] alg: No test for adler32 (adler32-zlib) [ 8678.055826] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 8678.330280] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8680.007135] Key type lgssc registered [ 8681.198153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8688.756516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8688.789662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.535807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.259660] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8699.884793] LustreError: 272829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.913700] LustreError: 272829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8704.909732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.994388] LustreError: 272830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.280058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.671923] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8718.246379] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8724.571582] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8732.122046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.513769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.596914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8740.359327] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8749.205611] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8775.659792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.661175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8775.691142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8775.708025] Lustre: Skipped 1 previous similar message [ 8780.494983] Lustre: server umount lustre-OST0000 complete [ 8785.889213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.903500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.908962] Lustre: Skipped 2 previous similar messages [ 8791.009719] LustreError: 272830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.028397] LustreError: 272830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8792.893210] Lustre: server umount lustre-MDT0000 complete [ 8795.168645] LustreError: 272829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.200774] LustreError: 272829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8797.331814] LustreError: 272811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834499 with bad export cookie 1389204195498580463 [ 8797.352993] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8797.624994] Lustre: server umount lustre-MDT0001 complete [ 8811.375416] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8814.879923] Key type lgssc unregistered [ 8815.199440] LNet: 277140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8815.221674] LNetError: 277140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8815.267513] LNet: Removed LNI 192.168.201.111@tcp [ 8816.135222] Key type .llcrypt unregistered [ 8816.137257] Key type ._llcrypt unregistered [ 8835.408066] Key type ._llcrypt registered [ 8835.410351] Key type .llcrypt registered [ 8835.512846] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8836.656972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8836.961256] alg: No test for adler32 (adler32-zlib) [ 8838.119961] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 8838.346681] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8840.135914] Key type lgssc registered [ 8841.460606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8849.861764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8849.875055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.335832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.618543] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8861.667844] LustreError: 278377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.701739] LustreError: 278377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8865.755442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.141340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8870.648240] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8878.703852] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8884.733391] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8892.451119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.939294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.062952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8899.221174] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8906.693947] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8921.581260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8921.584542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.595747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8922.592266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8922.594399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8922.603939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.637454] Lustre: Skipped 1 previous similar message [ 8926.649688] Lustre: server umount lustre-OST0000 complete [ 8932.836982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8932.838859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.856601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.876824] Lustre: Skipped 1 previous similar message [ 8936.948353] Lustre: server umount lustre-MDT0000 complete [ 8937.953171] LustreError: 278382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.971106] LustreError: 278382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8940.876549] LustreError: 278343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834643 with bad export cookie 12394093765079803426 [ 8940.886426] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8940.892587] LustreError: 278343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8941.145132] Lustre: server umount lustre-MDT0001 complete [ 8954.422440] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8957.586333] Key type lgssc unregistered [ 8957.847771] LNet: 282082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8957.853814] LNetError: 282082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8957.870423] LNet: Removed LNI 192.168.201.111@tcp [ 8958.837766] Key type .llcrypt unregistered [ 8958.840148] Key type ._llcrypt unregistered [ 8971.186472] Key type ._llcrypt registered [ 8971.189558] Key type .llcrypt registered [ 8971.325286] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8989.474474] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 9006.530897] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9007.447059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9007.560066] alg: No test for adler32 (adler32-zlib) [ 9008.707672] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9009.050924] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9010.815485] Key type lgssc registered [ 9012.074734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9048.889459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9048.908656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.142353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9050.197373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9050.272351] Lustre: lustre-MDT0000: new disk, initializing [ 9050.370613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.406577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9055.592397] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9067.543483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.695499] Lustre: 285636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9067.763965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9067.768361] Lustre: Skipped 1 previous similar message [ 9067.852972] Lustre: lustre-MDT0001: new disk, initializing [ 9068.051816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9068.103071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9068.127827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9073.749171] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9084.589566] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9090.757577] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9097.503906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9097.878044] Lustre: lustre-OST0000: new disk, initializing [ 9097.881877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9097.892620] Lustre: 287602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9097.990128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9104.660763] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9106.533151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9106.546127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9106.754503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9114.617797] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9119.854362] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9120.051660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9125.950925] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9126.307868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9132.512614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9132.517438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.524484] LustreError: Skipped 1 previous similar message [ 9132.527421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9132.550421] Lustre: Skipped 1 previous similar message [ 9134.795868] Lustre: server umount lustre-OST0000 complete [ 9139.689867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9139.701633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9139.728685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9139.731514] Lustre: Skipped 1 previous similar message [ 9142.752921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9142.769062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9145.478758] Lustre: server umount lustre-MDT0000 complete [ 9148.993913] LustreError: 285627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834851 with bad export cookie 13337303948384471334 [ 9149.005074] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9149.023780] LustreError: 285627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9149.389396] Lustre: server umount lustre-MDT0001 complete [ 9159.444516] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:00:59 (1786834859) [ 9166.040839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.408726] LustreError: 290025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.425782] LustreError: 290025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9166.505663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.255224] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9171.943574] LustreError: 290026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.033991] LustreError: 290025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.167070] LustreError: 290026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.627864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.076320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9187.164064] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9197.495834] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9203.740488] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9211.029049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9211.408667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.410414] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9226.511968] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9238.503919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9238.506619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.511694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.511705] Lustre: Skipped 1 previous similar message [ 9238.528386] Lustre: Skipped 1 previous similar message [ 9243.616635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9243.623358] Lustre: Skipped 1 previous similar message [ 9244.470403] Lustre: server umount lustre-OST0000 complete [ 9253.859683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9253.870789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9253.872396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9253.872404] Lustre: Skipped 1 previous similar message [ 9253.907662] Lustre: Skipped 1 previous similar message [ 9263.071802] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9263.391400] Lustre: server umount lustre-MDT0000 complete [ 9264.098029] LustreError: 290755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9264.117754] LustreError: 290755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9267.982791] LustreError: 292607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834970 with bad export cookie 13337303948384472216 [ 9267.991260] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.994678] LustreError: 292607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9268.312537] Lustre: server umount lustre-MDT0001 complete [ 9280.222934] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9283.065234] Key type lgssc unregistered [ 9283.359870] LNet: 293565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9283.365986] LNetError: 293565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9283.377170] LNet: Removed LNI 192.168.201.111@tcp [ 9284.214412] Key type .llcrypt unregistered [ 9284.219491] Key type ._llcrypt unregistered [ 9318.982748] Key type ._llcrypt registered [ 9318.986190] Key type .llcrypt registered [ 9319.129382] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 9338.446952] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9339.978035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9340.169360] alg: No test for adler32 (adler32-zlib) [ 9341.234295] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9341.536626] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9343.224204] Key type lgssc registered [ 9344.637102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9395.057770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9395.102622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.438171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9396.461973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9396.550523] Lustre: lustre-MDT0000: new disk, initializing [ 9396.607737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.630500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9401.214658] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9411.901771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.034673] Lustre: 297392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9412.107666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9412.111615] Lustre: Skipped 1 previous similar message [ 9412.174199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9412.223195] Lustre: lustre-MDT0001: new disk, initializing [ 9412.300494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9412.342237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9412.368478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9417.215547] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9427.206829] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9433.334316] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9440.323224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9440.532872] Lustre: lustre-OST2710: new disk, initializing [ 9440.536565] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9440.542050] Lustre: 299360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9440.619377] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9446.641741] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9450.637087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9450.648561] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9450.772043] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9456.424456] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9463.379368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9463.510400] Lustre: lustre-OST03e8: new disk, initializing [ 9463.513119] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9463.518048] Lustre: 300422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9463.589871] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9470.206668] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9472.658900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9472.666343] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9472.782063] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9478.204768] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9485.250897] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9485.531988] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9490.044200] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9490.286663] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9495.217222] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9495.446488] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9499.822312] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9500.045172] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9504.737245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9504.743913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.753488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9506.314264] Lustre: server umount lustre-MDT0000 complete [ 9508.837424] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.909962] LustreError: 297386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835212 with bad export cookie 8645678614393671490 [ 9509.928585] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9510.243808] Lustre: server umount lustre-MDT0001 complete [ 9524.274516] Lustre: server umount lustre-OST2710 complete [ 9537.706488] Lustre: server umount lustre-OST03e8 complete [ 9553.540623] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 9563.181244] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9598.140276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.457559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9598.497446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9598.625868] Lustre: lustre-MDT0000: new disk, initializing [ 9598.807659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.842106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9603.782586] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9615.032495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.162490] Lustre: 305672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9615.257376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9615.262385] Lustre: Skipped 1 previous similar message [ 9615.342659] Lustre: lustre-MDT0001: new disk, initializing [ 9615.442265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.494499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9615.509845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9620.128854] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9630.244580] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9636.032912] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9643.140297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9643.507625] Lustre: lustre-OST0000: new disk, initializing [ 9643.515466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9643.522918] Lustre: 307641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9643.596545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9650.250900] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9651.292191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9651.300437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9651.446739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9659.932349] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9664.484061] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9664.706710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9670.082646] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9670.302733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9677.283515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9677.284674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9677.299848] Lustre: Skipped 2 previous similar messages [ 9677.305810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9677.328874] Lustre: Skipped 1 previous similar message [ 9678.642840] Lustre: server umount lustre-OST0000 complete [ 9687.013401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9687.029774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.054210] Lustre: Skipped 1 previous similar message [ 9687.068798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9689.212957] Lustre: server umount lustre-MDT0000 complete [ 9692.643119] LustreError: 305680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.674937] LustreError: 305680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9693.153591] LustreError: 305665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835395 with bad export cookie 8645678614393673170 [ 9693.166435] LustreError: 305665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9693.168399] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9693.484443] Lustre: server umount lustre-MDT0001 complete [ 9702.407208] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:10:02 (1786835402) [ 9704.046933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9706.095879] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:10:06 (1786835406) [ 9709.855561] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9737.291185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.589577] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9737.607567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9737.748725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.936961] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9750.784876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.856740] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9750.893700] Lustre: 311500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9755.065307] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9776.041642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.155668] Lustre: Denying initial registration attempt from nid 192.168.201.111@tcp, specified as failover [ 9776.159795] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9776.165425] LustreError: 312955:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9776.169486] LustreError: 312955:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9776.171727] LustreError: 312955:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9776.226917] Lustre: server umount lustre-OST0000 complete [ 9776.229267] LustreError: 312955:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9783.776681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.781088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9783.786099] Lustre: Skipped 2 previous similar messages [ 9783.810814] Lustre: Skipped 2 previous similar messages [ 9788.778897] Lustre: server umount lustre-MDT0000 complete [ 9788.897540] LustreError: 311508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.912978] LustreError: 311508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9792.556504] LustreError: 311492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835494 with bad export cookie 8645678614393674045 [ 9792.565308] LustreError: 311492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9792.572879] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9792.986166] Lustre: server umount lustre-MDT0001 complete [ 9802.588379] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9805.093829] Key type lgssc unregistered [ 9805.307803] LNet: 314077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9805.312519] LNetError: 314077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9805.324893] LNet: Removed LNI 192.168.201.111@tcp [ 9806.119785] Key type .llcrypt unregistered [ 9806.122567] Key type ._llcrypt unregistered [ 9816.868334] Key type ._llcrypt registered [ 9816.871380] Key type .llcrypt registered [ 9817.048796] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9818.166223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9818.184512] alg: No test for adler32 (adler32-zlib) [ 9819.190944] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9819.446709] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9821.143291] Key type lgssc registered [ 9822.107866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9866.420943] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9874.157173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9874.171430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9875.369106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9875.384492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9875.498909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.062407] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9885.719050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.793198] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9885.821853] Lustre: 316840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9886.046803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9890.033669] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9897.302538] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9902.023239] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9920.620076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.885613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9921.060597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9925.787348] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9932.320239] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9937.381238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9937.385092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9937.399290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9937.896192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9937.909775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9937.922867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9940.369729] Lustre: server umount lustre-OST0000 complete [ 9947.616555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9947.629963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9947.651608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.249758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.259679] Lustre: Skipped 3 previous similar messages [ 9958.369258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9958.370847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9958.389616] Lustre: Skipped 1 previous similar message [ 9958.587843] Lustre: server umount lustre-MDT0000 complete [ 9962.294041] LustreError: 316833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835664 with bad export cookie 3803166549651897788 [ 9962.306495] LustreError: 316833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9962.314784] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9962.645512] Lustre: server umount lustre-MDT0001 complete [ 9972.926850] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9975.509291] Key type lgssc unregistered [ 9975.834703] LNet: 320417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9975.848136] LNetError: 320417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9975.873815] LNet: Removed LNI 192.168.201.111@tcp [ 9976.708501] Key type .llcrypt unregistered [ 9976.710440] Key type ._llcrypt unregistered [ 9996.371981] Key type ._llcrypt registered [ 9996.373868] Key type .llcrypt registered [ 9996.456556] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9997.316737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9997.392365] alg: No test for adler32 (adler32-zlib) [ 9998.392624] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [ 9998.642215] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10000.343519] Key type lgssc registered [10001.586527] Lustre: Echo OBD driver; http://www.lustre.org/ [10008.424492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10008.451727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.759584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10013.271624] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10019.300685] LustreError: 321814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.319704] LustreError: 321814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10021.458676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.807370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10025.663930] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10031.945134] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10037.431508] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10045.674407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10046.096538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10050.981557] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10057.945175] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10067.936183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10067.950499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.962879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10069.988112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10069.989056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.014675] Lustre: Skipped 1 previous similar message [10070.025814] Lustre: Skipped 2 previous similar messages [10072.035287] Lustre: server umount lustre-MDT0000 complete [10075.106229] LustreError: 321814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.115641] LustreError: 321814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10075.535445] LustreError: 321793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835777 with bad export cookie 6425791975414132907 [10075.537702] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.546048] LustreError: 321793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10075.899611] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [10075.909449] Lustre: Skipped 1 previous similar message [10080.225256] LustreError: 321809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.225435] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10080.235253] LustreError: 321809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10080.251319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10081.968161] Lustre: server umount lustre-MDT0001 complete [10087.349629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10095.965188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10101.927611] LustreError: 323646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10101.947632] LustreError: 323646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10102.006365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10102.070580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10105.956487] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10106.023899] LustreError: 323647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10106.037977] LustreError: 323647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10110.119566] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10112.637080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.969758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10116.354083] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10120.169966] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10122.437685] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10123.792790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10128.078942] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10129.612980] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10135.532481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.534649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10135.552150] Lustre: Skipped 1 previous similar message [10135.558838] Lustre: Skipped 2 previous similar messages [10139.793329] Lustre: server umount lustre-OST0000 complete [10143.714257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10143.721173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.737281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10148.933583] Lustre: server umount lustre-MDT0000 complete [10150.880734] LustreError: 325413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10150.891689] LustreError: 325413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10152.093189] LustreError: 321793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835854 with bad export cookie 6425791975414140754 [10152.107335] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.112408] LustreError: 321793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10152.444412] Lustre: server umount lustre-MDT0001 complete [10161.075466] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10163.171607] Key type lgssc unregistered [10163.386677] LNet: 328083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10163.397813] LNetError: 328083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10163.412772] LNet: Removed LNI 192.168.201.111@tcp [10163.991137] Key type .llcrypt unregistered [10163.992546] Key type ._llcrypt unregistered [10182.364404] Key type ._llcrypt registered [10182.366430] Key type .llcrypt registered [10182.437401] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10183.229043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10183.323101] alg: No test for adler32 (adler32-zlib) [10184.348288] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10184.563564] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10186.223166] Key type lgssc registered [10186.960843] Lustre: Echo OBD driver; http://www.lustre.org/ [10192.881946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10192.895492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10194.230266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10197.677995] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10203.888886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10204.145638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10207.195582] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10212.397486] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10216.250366] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10219.488136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10219.493431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.506966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10219.508317] Lustre: Skipped 1 previous similar message [10219.523120] Lustre: Skipped 1 previous similar message [10224.612103] LustreError: 329483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10224.618719] LustreError: 329483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10224.645623] Lustre: server umount lustre-MDT0000 complete [10226.785206] LustreError: 329470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835928 with bad export cookie 11652784181684308352 [10226.787302] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10226.791475] LustreError: 329470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10226.966627] Lustre: server umount lustre-MDT0001 complete [10235.981936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.149949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10236.167674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10236.315539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10239.281680] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10245.410854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.459859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10245.481174] Lustre: 332074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10245.670974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10248.542043] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10253.796565] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10257.826188] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10264.998452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10265.151665] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10265.159138] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10265.165451] LustreError: 333957:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10265.169699] LustreError: 333957:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10265.173102] LustreError: 333957:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10265.222825] Lustre: server umount lustre-OST0000 complete [10265.225255] LustreError: 333957:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10273.330661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10273.498898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10273.640564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10277.647140] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10279.483893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10282.451141] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10287.567278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10287.625988] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10287.668846] Lustre: lustre-OST0001: new disk, initializing [10287.670814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10287.673702] Lustre: 335496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10287.739212] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10291.818086] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10295.374977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10295.391959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10295.452545] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10297.623345] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10300.901403] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10300.913983] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10300.914061] LustreError: Skipped 1 previous similar message [10300.924687] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10300.927737] Lustre: Skipped 1 previous similar message [10301.793983] Lustre: server umount lustre-OST0001 complete [10304.992952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10304.993970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.997679] LustreError: Skipped 1 previous similar message [10304.999028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10305.006797] Lustre: Skipped 2 previous similar messages [10305.019342] Lustre: Skipped 1 previous similar message [10306.024825] LustreError: 334422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10306.037042] LustreError: 334422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10310.859512] Lustre: server umount lustre-OST0000 complete [10316.258148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10316.258845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10316.272135] Lustre: Skipped 1 previous similar message [10316.277161] Lustre: Skipped 3 previous similar messages [10319.526140] Lustre: server umount lustre-MDT0000 complete [10321.377305] LustreError: 333076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10321.396908] LustreError: 333076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10322.505042] LustreError: 332069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836024 with bad export cookie 11652784181684308905 [10322.507656] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.509815] LustreError: 332069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10322.704755] Lustre: server umount lustre-MDT0001 complete [10330.539495] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10332.395905] Key type lgssc unregistered [10332.621575] LNet: 337458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10332.628555] LNetError: 337458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10332.644973] LNet: Removed LNI 192.168.201.111@tcp [10333.227082] Key type .llcrypt unregistered [10333.228631] Key type ._llcrypt unregistered [10386.810461] Key type ._llcrypt registered [10386.812776] Key type .llcrypt registered [10386.903684] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [10398.163329] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10399.002575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10399.036152] alg: No test for adler32 (adler32-zlib) [10400.030970] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10400.210176] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10401.879192] Key type lgssc registered [10402.804899] Lustre: Echo OBD driver; http://www.lustre.org/ [10427.705033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10427.731660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.930520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10428.963588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10429.038868] Lustre: lustre-MDT0000: new disk, initializing [10429.108836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10429.125627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10432.090574] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10439.963557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.091334] Lustre: 342367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10440.140442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10440.143967] Lustre: Skipped 1 previous similar message [10440.265324] Lustre: lustre-MDT0001: new disk, initializing [10440.368489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10440.418806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10440.430751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10443.363867] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10450.508335] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10454.604550] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10459.798571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10459.975845] Lustre: lustre-OST0000: new disk, initializing [10459.978854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10459.983447] Lustre: 344334:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10460.057293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10464.442303] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10465.337626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10465.345215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10465.398722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10470.984356] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10473.945717] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10474.099081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10476.652429] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10476.766257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10480.616062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10480.620500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10480.625542] Lustre: Skipped 1 previous similar message [10485.731812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10485.735964] Lustre: Skipped 2 previous similar messages [10490.850721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10490.856910] Lustre: Skipped 1 previous similar message [10492.384871] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10492.499795] Lustre: server umount lustre-OST0000 complete [10495.969606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10495.970967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10495.976806] Lustre: Skipped 1 previous similar message [10501.089290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10501.092194] Lustre: Skipped 2 previous similar messages [10509.279136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10509.418631] Lustre: server umount lustre-MDT0000 complete [10512.181164] LustreError: 342358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836214 with bad export cookie 6671487922350393388 [10512.184549] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10512.190419] LustreError: 342358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10512.470153] Lustre: server umount lustre-MDT0001 complete [10518.486483] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:23:39 (1786836219) [10523.641081] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:23:44 (1786836224) [10528.741760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.978799] LustreError: 347089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10528.989388] LustreError: 347089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10529.020912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10531.379609] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10534.369608] LustreError: 347090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10534.392851] LustreError: 347090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10536.888956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.096897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10540.380880] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10545.011059] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10548.550223] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10552.309672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10552.528487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10555.823457] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10560.349729] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10569.478444] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10570.508092] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10571.443198] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10572.502214] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10573.888470] Lustre: Failing over lustre-MDT0000 [10574.081564] Lustre: server umount lustre-MDT0000 complete [10577.117778] LustreError: 347085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10578.401311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10578.403700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10578.406069] Lustre: Skipped 2 previous similar messages [10582.230049] LustreError: 347084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.237994] LustreError: 347084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10587.348498] LustreError: 347084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.359150] LustreError: 347084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10588.307226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.397301] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.586907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10588.623404] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10590.563864] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10592.426018] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10593.774226] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10593.793644] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10593.815531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10596.445043] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10597.281176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10599.668373] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10600.725105] Lustre: Failing over lustre-MDT0000 [10600.898637] Lustre: server umount lustre-MDT0000 complete [10602.725658] LustreError: 347084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.004402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10604.004624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10604.022461] Lustre: Skipped 1 previous similar message [10615.970166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.072918] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.224679] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10618.072596] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10618.393755] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10621.413592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10621.416459] Lustre: Skipped 2 previous similar messages [10621.434995] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10621.463832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10624.564882] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10625.458747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10631.649578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10631.653443] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.659406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10631.663343] Lustre: Skipped 2 previous similar messages [10631.669173] Lustre: Skipped 4 previous similar messages [10635.186070] Lustre: server umount lustre-MDT0000 complete [10636.770129] LustreError: 347086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10636.780671] LustreError: 347086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [10637.146737] LustreError: 351048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836339 with bad export cookie 6671487922350396083 [10637.149038] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10637.157723] LustreError: 351048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10637.315235] Lustre: server umount lustre-MDT0001 complete [10649.600688] Lustre: server umount lustre-OST0000 complete [10657.654985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10657.987666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10657.990488] Lustre: Skipped 1 previous similar message [10659.818131] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10664.266403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.335387] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10669.780300] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10672.324524] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10675.220489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10677.782923] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10680.955737] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10683.567272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10694.112302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10694.116323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10694.120539] Lustre: Skipped 1 previous similar message [10694.122642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10699.614668] Lustre: server umount lustre-OST0000 complete [10705.375948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10705.378542] LustreError: Skipped 1 previous similar message [10707.372135] Lustre: server umount lustre-MDT0000 complete [10708.982820] LustreError: 353400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836411 with bad export cookie 6671487922350396832 [10708.983775] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10708.988118] LustreError: 353400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10709.142635] Lustre: server umount lustre-MDT0001 complete [10713.679380] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10714.878695] Key type lgssc unregistered [10715.020769] LNet: 356971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10715.024150] LNetError: 356971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10715.036348] LNet: Removed LNI 192.168.201.111@tcp [10715.397924] Key type .llcrypt unregistered [10715.399796] Key type ._llcrypt unregistered [10727.616955] Key type ._llcrypt registered [10727.617930] Key type .llcrypt registered [10727.663938] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10728.155115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10728.165241] alg: No test for adler32 (adler32-zlib) [10729.047547] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10729.163641] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10730.767170] Key type lgssc registered [10731.348921] Lustre: Echo OBD driver; http://www.lustre.org/ [10734.689152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10734.703790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.896186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10737.523089] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10741.267450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10741.422033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10743.130378] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10746.233499] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10748.658801] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10751.673640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.865334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10754.491733] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10757.501876] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10761.064116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10772.448022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10772.450638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10772.456865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.545316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.546980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10778.152789] Lustre: server umount lustre-MDT0000 complete [10779.735531] LustreError: 358444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836481 with bad export cookie 5862618471879059613 [10779.740237] LustreError: 358444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10779.740606] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10779.917314] Lustre: server umount lustre-MDT0001 complete [10791.912143] Lustre: server umount lustre-OST0000 complete [10797.170313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.337837] LustreError: 362192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.343120] LustreError: 362192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10797.366676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.839704] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10802.047529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.190964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10803.774489] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10806.699460] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10808.907472] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10811.498798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.668511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10814.017981] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10817.338715] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10822.950607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10837.983899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10837.986904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10837.990642] Lustre: Skipped 1 previous similar message [10837.992767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10837.994894] Lustre: Skipped 2 previous similar messages [10841.121853] Lustre: server umount lustre-MDT0000 complete [10842.636325] LustreError: 362173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836544 with bad export cookie 5862618471879060712 [10842.638985] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10842.640503] LustreError: 362173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10842.769321] Lustre: server umount lustre-MDT0001 complete [10854.693161] Lustre: server umount lustre-OST0000 complete [10861.482114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.673943] LustreError: 366126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.706255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10863.270710] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10866.747892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.663282] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10871.927254] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10874.247262] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10876.897637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10879.199459] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10882.011921] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10886.249365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10891.425626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10891.425847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10891.428942] Lustre: Skipped 2 previous similar messages [10891.430462] Lustre: Skipped 3 previous similar messages [10894.490795] Lustre: server umount lustre-OST0000 complete [10896.864216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.865245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.868906] Lustre: Skipped 1 previous similar message [10896.871545] Lustre: Skipped 1 previous similar message [10902.330124] Lustre: server umount lustre-MDT0000 complete [10904.114868] LustreError: 366106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836606 with bad export cookie 5862618471879061720 [10904.115680] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.119798] LustreError: 366106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10904.261050] Lustre: server umount lustre-MDT0001 complete [10908.927922] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10910.161238] Key type lgssc unregistered [10910.306687] LNet: 369655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10910.309799] LNetError: 369655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10910.322435] LNet: Removed LNI 192.168.201.111@tcp [10910.657802] Key type .llcrypt unregistered [10910.659064] Key type ._llcrypt unregistered [10922.611576] Key type ._llcrypt registered [10922.612705] Key type .llcrypt registered [10922.655906] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10923.165840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10923.173875] alg: No test for adler32 (adler32-zlib) [10924.054088] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10924.154045] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10925.743150] Key type lgssc registered [10926.299667] Lustre: Echo OBD driver; http://www.lustre.org/ [10929.908242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10929.916772] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.920142] LustreError: 371072:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10929.932515] LustreError: 371072:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10929.937452] LustreError: 371072:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10929.940591] LustreError: 371072:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10929.943458] LustreError: 371072:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10933.776352] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10933.784157] LustreError: 371469:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10933.799592] LustreError: 371469:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10933.806388] LustreError: 371469:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10933.809316] LustreError: 371469:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10933.812224] LustreError: 371469:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10943.502224] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [10944.891934] Key type lgssc unregistered [10945.067348] LNet: 372474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10945.070536] LNetError: 372474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10945.079450] LNet: Removed LNI 192.168.201.111@tcp [10945.497603] Key type .llcrypt unregistered [10945.498631] Key type ._llcrypt unregistered [10954.690653] Key type ._llcrypt registered [10954.692315] Key type .llcrypt registered [10954.749797] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [10963.419251] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10963.946501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10964.010860] alg: No test for adler32 (adler32-zlib) [10964.926854] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [10965.074126] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [10966.679167] Key type lgssc registered [10967.263393] Lustre: Echo OBD driver; http://www.lustre.org/ [10984.004515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10984.012173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10985.141460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10985.154252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10985.194774] Lustre: lustre-MDT0000: new disk, initializing [10985.235872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10985.245053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10987.143499] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10992.511801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.564959] Lustre: 375602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10992.599334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10992.601992] Lustre: Skipped 1 previous similar message [10992.642485] Lustre: lustre-MDT0001: new disk, initializing [10992.690830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10992.711024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10992.714979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10994.677994] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10999.879818] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11002.549294] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11006.088243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11006.225645] Lustre: lustre-OST0000: new disk, initializing [11006.228381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11006.232556] Lustre: 377572:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11006.280194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11007.311343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11007.314682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11007.379824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11008.980641] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11013.799403] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11015.741467] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11015.831145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11017.545284] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11017.652562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11022.820789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.821299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11022.825679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11022.827125] LustreError: Skipped 1 previous similar message [11022.831415] Lustre: Skipped 1 previous similar message [11024.665437] Lustre: server umount lustre-OST0000 complete [11027.936688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11027.937506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11027.945291] Lustre: Skipped 2 previous similar messages [11032.371807] Lustre: server umount lustre-MDT0000 complete [11034.145593] LustreError: 375597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836736 with bad export cookie 3948294229001149555 [11034.147250] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11034.150802] LustreError: 375597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11034.306963] Lustre: server umount lustre-MDT0001 complete [11038.392684] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11039.222220] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:32:20 (1786836740) [11042.576845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.828744] LustreError: 380026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.840061] LustreError: 380026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11042.872877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11045.054682] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11047.907053] LustreError: 380027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.060726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.201816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11051.226279] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11054.891083] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11057.477534] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11060.450833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11060.593062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11063.570627] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11066.917276] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11069.978804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11070.053422] Lustre: lustre-OST0001: new disk, initializing [11070.056401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11070.059800] Lustre: 382857:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11072.613960] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11077.205575] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11080.205677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11080.208463] Lustre: Skipped 1 previous similar message [11083.487583] Lustre: lustre-OST0001: Not available for connect from 192.168.201.11@tcp (stopping) [11083.493277] Lustre: Skipped 1 previous similar message [11085.437440] Lustre: server umount lustre-OST0001 complete [11090.403649] LustreError: 383659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.409255] LustreError: 383659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11095.525651] LustreError: 381843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.534854] LustreError: 381843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11098.592541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11098.596040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11098.601435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11098.603618] Lustre: Skipped 2 previous similar messages [11100.128281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11100.642269] LustreError: 383659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.648652] LustreError: 383659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11104.019383] Lustre: server umount lustre-OST0000 complete [11105.760832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11105.764036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.768117] Lustre: Skipped 2 previous similar messages [11110.880259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11110.883353] Lustre: Skipped 5 previous similar messages [11111.853063] Lustre: server umount lustre-MDT0000 complete [11113.485093] LustreError: 380007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836815 with bad export cookie 3948294229001150437 [11113.488644] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.488762] LustreError: 380007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11119.711832] Lustre: server umount lustre-MDT0001 complete [11124.365053] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11125.666239] Key type lgssc unregistered [11125.824671] LNet: 384845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11125.829670] LNetError: 384845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11125.844336] LNet: Removed LNI 192.168.201.111@tcp [11126.230497] Key type .llcrypt unregistered [11126.231555] Key type ._llcrypt unregistered [11149.950570] Key type ._llcrypt registered [11149.951699] Key type .llcrypt registered [11149.994705] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11150.353498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11150.390269] alg: No test for adler32 (adler32-zlib) [11151.270337] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11151.383229] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11152.983156] Key type lgssc registered [11153.412100] Lustre: Echo OBD driver; http://www.lustre.org/ [11157.097135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11157.103365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11158.210421] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11158.219424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11158.283984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11160.323449] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11164.700025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11164.789956] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11164.810837] Lustre: 387619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11164.921913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11166.882228] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11170.589878] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11173.270773] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11175.392736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11175.399384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11175.406660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11175.906106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11175.913593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11180.416304] Lustre: server umount lustre-MDT0000 complete [11182.300386] LustreError: 387612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836884 with bad export cookie 8125500919337623301 [11182.303266] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11182.309417] LustreError: 387612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11182.432535] Lustre: server umount lustre-MDT0001 complete [11187.285763] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:34:48 (1786836888) [11190.504931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.758582] LustreError: 390128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11190.765676] LustreError: 390128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11190.790902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11192.630423] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11195.874196] LustreError: 390129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.797199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11197.024273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11198.769514] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11202.264691] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11204.907683] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11207.977407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11208.081990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11208.187423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11210.497785] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11213.168526] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11217.098304] Lustre: Permanently deactivating lustre-OST0000 [11217.104585] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11222.520428] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11223.347218] LustreError: 393111:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11223.350532] LustreError: 393111:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11223.352761] LustreError: 393111:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11224.143568] LustreError: 393160:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11224.146835] LustreError: 393160:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11231.077609] Lustre: server umount lustre-OST0000 complete [11232.273447] LustreError: 393409:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11232.276159] LustreError: 393409:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11233.760952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11233.762827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11233.767378] Lustre: Skipped 1 previous similar message [11238.880260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11238.883727] Lustre: Skipped 2 previous similar messages [11243.013404] Lustre: server umount lustre-MDT0000 complete [11244.000330] LustreError: 390123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.011871] LustreError: 390123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11244.626392] LustreError: 392816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836946 with bad export cookie 8125500919337623868 [11244.626933] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11244.630077] LustreError: 392816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11254.308065] Lustre: server umount lustre-MDT0001 complete [11258.435673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.161748] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11261.393754] LustreError: 394567:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11264.140599] Lustre: 394906:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.111@tcp [11265.119800] Lustre: server umount MGS complete [11270.084256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11270.342730] LustreError: 395628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.350987] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11270.354678] Lustre: Skipped 1 previous similar message [11270.385967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11272.209734] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11275.744099] LustreError: 395629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.018634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11276.155616] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11277.985902] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11281.248170] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11283.584000] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11284.514423] Lustre: Permanently reactivating lustre-OST0000 [11284.522197] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11284.524334] Lustre: Skipped 2 previous similar messages [11306.465445] LustreError: 397311:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11309.178966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11309.314778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11309.317034] Lustre: Skipped 1 previous similar message [11311.418563] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11314.192431] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11326.945875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.946093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11326.951546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11326.952982] LustreError: Skipped 1 previous similar message [11328.342131] Lustre: server umount lustre-OST0000 complete [11331.869315] Lustre: server umount lustre-MDT0000 complete [11332.064356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11332.066864] LustreError: 395647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11332.075047] Lustre: Skipped 2 previous similar messages [11333.319145] LustreError: 395610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837035 with bad export cookie 8125500919337624904 [11333.321502] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11333.322805] LustreError: 395610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11333.459052] Lustre: server umount lustre-MDT0001 complete [11337.387748] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11338.520862] Key type lgssc unregistered [11338.678713] LNet: 400109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11338.682630] LNetError: 400109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11338.689268] LNet: Removed LNI 192.168.201.111@tcp [11339.064858] Key type .llcrypt unregistered [11339.065901] Key type ._llcrypt unregistered [11346.276691] Key type ._llcrypt registered [11346.277983] Key type .llcrypt registered [11346.328694] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [11353.697561] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11354.050069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11354.082071] alg: No test for adler32 (adler32-zlib) [11354.994725] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11355.106468] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11356.719179] Key type lgssc registered [11357.169768] Lustre: Echo OBD driver; http://www.lustre.org/ [11371.097435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11371.105868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.211748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11372.225381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11372.265251] Lustre: lustre-MDT0000: new disk, initializing [11372.291046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11372.298500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11373.871723] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11378.792457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.836714] Lustre: 403238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11378.864747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11378.867046] Lustre: Skipped 1 previous similar message [11378.910588] Lustre: lustre-MDT0001: new disk, initializing [11378.944737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11378.960204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11378.964847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11380.400057] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11384.683616] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11386.834421] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11389.387467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11389.524910] Lustre: lustre-OST0000: new disk, initializing [11389.526975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11389.529917] Lustre: 405206:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11389.569790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11391.811661] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11396.048593] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11397.638489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11397.642343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11397.684861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11397.729541] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11397.830173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11399.425211] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11399.513114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11402.721684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11402.725861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11402.726657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11402.729527] LustreError: Skipped 1 previous similar message [11402.731447] Lustre: Skipped 1 previous similar message [11406.432883] Lustre: server umount lustre-OST0000 complete [11409.888157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11409.891297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.895817] Lustre: Skipped 1 previous similar message [11409.898626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11412.960306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11412.960892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11418.080837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11418.085248] Lustre: Skipped 2 previous similar messages [11422.175113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11422.176550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11422.179113] Lustre: Skipped 1 previous similar message [11422.234224] Lustre: server umount lustre-MDT0000 complete [11423.695585] LustreError: 403231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837125 with bad export cookie 3012924606602061277 [11423.696550] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11423.702024] LustreError: 403231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11423.833832] Lustre: server umount lustre-MDT0001 complete [11427.220298] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11428.016529] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:38:49 (1786837129) [11430.834922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.085321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11432.657463] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11436.102606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.258609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11437.943084] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11440.863372] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11443.345910] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11446.671654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11446.827379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11449.250332] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11452.277688] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11453.115538] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11459.904224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11471.329068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11471.330580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11471.331165] LustreError: Skipped 1 previous similar message [11471.331986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11471.336274] Lustre: Skipped 1 previous similar message [11471.342242] Lustre: Skipped 1 previous similar message [11481.055378] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11481.118892] Lustre: server umount lustre-OST0000 complete [11486.688534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11486.695075] Lustre: Skipped 1 previous similar message [11488.796691] Lustre: server umount lustre-MDT0000 complete [11490.271708] LustreError: 407644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837192 with bad export cookie 3012924606602062138 [11490.274650] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11490.277580] LustreError: 407644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11490.415566] Lustre: server umount lustre-MDT0001 complete [11494.553792] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11495.740403] Key type lgssc unregistered [11495.891341] LNet: 411412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11495.894540] LNetError: 411412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11495.903351] LNet: Removed LNI 192.168.201.111@tcp [11496.239702] Key type .llcrypt unregistered [11496.241047] Key type ._llcrypt unregistered [11505.914095] Key type ._llcrypt registered [11505.915857] Key type .llcrypt registered [11505.959460] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11506.096326] Key type .llcrypt unregistered [11506.098227] Key type ._llcrypt unregistered [11514.241570] Key type ._llcrypt registered [11514.243015] Key type .llcrypt registered [11514.280352] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11514.757497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11514.766821] alg: No test for adler32 (adler32-zlib) [11515.641350] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11515.753464] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11517.361466] Key type lgssc registered [11517.890397] Lustre: Echo OBD driver; http://www.lustre.org/ [11521.175601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11521.185318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11522.423741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11524.036754] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11527.690906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11527.822511] LustreError: 413600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11527.828894] LustreError: 413600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11527.829330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.361123] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11531.937578] LustreError: 413604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.941730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11531.944750] LustreError: 413604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11533.123361] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11535.672171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11535.827321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11537.339324] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11546.082113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11546.083112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.084259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11546.092585] Lustre: Skipped 1 previous similar message [11550.626698] Lustre: server umount lustre-OST0000 complete [11554.787231] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11554.790374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11554.795963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11554.799648] Lustre: Skipped 1 previous similar message [11556.320638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11556.322381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.228743] Lustre: server umount lustre-MDT0000 complete [11559.632150] LustreError: 413584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837261 with bad export cookie 8022142306270598587 [11559.636333] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11559.636338] LustreError: 413584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11559.762816] Lustre: server umount lustre-MDT0001 complete [11564.392185] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11565.658057] Key type lgssc unregistered [11565.801990] LNet: 416752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11565.804227] LNetError: 416752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11565.811514] LNet: Removed LNI 192.168.201.111@tcp [11566.235241] Key type .llcrypt unregistered [11566.236343] Key type ._llcrypt unregistered [11577.002364] Key type ._llcrypt registered [11577.003344] Key type .llcrypt registered [11577.043146] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11577.483740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11577.497867] alg: No test for adler32 (adler32-zlib) [11578.359985] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11578.460540] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11580.047127] Key type lgssc registered [11580.447599] Lustre: Echo OBD driver; http://www.lustre.org/ [11583.308311] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11583.315773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11598.751396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11603.871318] LustreError: 418118:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.111@tcp: failed processing log, type 4: rc = -110 [11630.495193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11632.821680] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11635.705901] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11638.429949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11638.628701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11638.664090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11640.193833] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11643.455610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.612566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11645.184111] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11648.076907] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11650.234606] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11654.112487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11654.112932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11654.115024] LustreError: Skipped 1 previous similar message [11654.120863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11654.124609] Lustre: Skipped 1 previous similar message [11656.162828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11656.165329] Lustre: Skipped 1 previous similar message [11659.999576] Lustre: server umount lustre-OST0000 complete [11663.841696] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11663.844186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.849062] Lustre: Skipped 1 previous similar message [11663.854276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11666.400069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11666.400670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11671.520055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11671.522550] Lustre: Skipped 2 previous similar messages [11677.151134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11677.221724] Lustre: server umount lustre-MDT0000 complete [11678.619805] LustreError: 420644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837380 with bad export cookie 13487967873829718739 [11678.620526] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11678.626078] LustreError: 420644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11678.741563] Lustre: server umount lustre-MDT0001 complete [11683.056159] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11684.199026] Key type lgssc unregistered [11684.344418] LNet: 421689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.346813] LNetError: 421689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.355433] LNet: Removed LNI 192.168.201.111@tcp [11684.700937] Key type .llcrypt unregistered [11684.702477] Key type ._llcrypt unregistered [11695.836863] Key type ._llcrypt registered [11695.838546] Key type .llcrypt registered [11695.884121] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11696.342926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11696.356730] alg: No test for adler32 (adler32-zlib) [11697.237268] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11697.367105] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11698.959238] Key type lgssc registered [11699.459081] Lustre: Echo OBD driver; http://www.lustre.org/ [11702.618922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11702.626050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.803334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11705.369322] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11708.829443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11709.015949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11710.678940] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11713.614681] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11715.794275] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11718.301079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11718.441932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11720.612360] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11723.373952] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11728.678742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11738.843741] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [11739.105602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.110189] Lustre: Skipped 2 previous similar messages [11740.756465] Lustre: server umount lustre-MDT0000 complete [11741.524199] LustreError: 423064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.11@tcp arrived at 1786837443 with bad export cookie 13709517560533550589 [11741.529620] LustreError: 423064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11742.207526] LustreError: 424926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837444 with bad export cookie 13709517560533549966 [11742.211625] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11742.215341] LustreError: 424926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11744.224041] LustreError: 423082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.224776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11744.230365] LustreError: 423082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11744.240081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11744.241976] Lustre: Skipped 3 previous similar messages [11747.232025] LustreError: 423823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.232792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11747.239286] LustreError: 423823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11748.311467] Lustre: server umount lustre-OST0000 complete [11750.825806] Lustre: server umount lustre-MDT0001 complete [11754.832833] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11756.019754] Key type lgssc unregistered [11756.150479] LNet: 426786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11756.153714] LNetError: 426786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11756.164469] LNet: Removed LNI 192.168.201.111@tcp [11756.533483] Key type .llcrypt unregistered [11756.534747] Key type ._llcrypt unregistered [11766.936148] Key type ._llcrypt registered [11766.937547] Key type .llcrypt registered [11766.981091] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11767.324591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11767.334828] alg: No test for adler32 (adler32-zlib) [11768.213786] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11768.315297] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11769.911124] Key type lgssc registered [11770.294783] Lustre: Echo OBD driver; http://www.lustre.org/ [11772.996534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11773.004731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11774.163094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11775.566360] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11778.627150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11778.756301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11780.185335] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11782.917312] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11785.073075] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11787.916779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11788.004642] LustreError: 430011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.012990] LustreError: 430011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11788.086804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11790.334280] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11793.226289] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11799.526653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11804.645741] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11804.649418] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.655288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11804.659852] Lustre: Skipped 1 previous similar message [11806.811563] Lustre: server umount lustre-MDT0001 complete [11809.760409] LustreError: 428194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.761174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.761696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11809.775161] Lustre: Skipped 1 previous similar message [11809.778722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11809.780979] Lustre: Skipped 1 previous similar message [11814.871865] Lustre: server umount lustre-OST0000 complete [11814.880930] LustreError: 428194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.331971] Lustre: server umount lustre-MDT0000 complete [11821.852319] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11822.954502] Key type lgssc unregistered [11823.095517] LNet: 431879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11823.098277] LNetError: 431879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11823.105969] LNet: Removed LNI 192.168.201.111@tcp [11823.454403] Key type .llcrypt unregistered [11823.455416] Key type ._llcrypt unregistered [11832.382733] Key type ._llcrypt registered [11832.383815] Key type .llcrypt registered [11832.427583] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [11838.981572] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11839.371131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11839.400121] alg: No test for adler32 (adler32-zlib) [11840.277489] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11840.391054] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11841.983199] Key type lgssc registered [11842.425127] Lustre: Echo OBD driver; http://www.lustre.org/ [11856.066801] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11861.716718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11861.725473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.817493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11862.829225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11862.866399] Lustre: lustre-MDT0000: new disk, initializing [11862.892340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.900118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11864.505771] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11870.808343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11870.912154] Lustre: lustre-OST0000: new disk, initializing [11870.914343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11870.916688] Lustre: Skipped 1 previous similar message [11870.918686] Lustre: 436883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11870.947679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11873.060266] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11875.847426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11875.851666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11875.861907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11877.047798] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11883.823778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11883.872482] Lustre: 435863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11883.899888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11883.936459] Lustre: lustre-MDT0001: new disk, initializing [11883.981096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11883.997449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11884.001908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11885.574194] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11889.549505] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11889.639529] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11891.258214] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11891.356831] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11895.266635] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11895.267154] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11895.271958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11895.275799] Lustre: Skipped 1 previous similar message [11896.538735] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.11@tcp (stopping) [11899.218323] Lustre: server umount lustre-MDT0001 complete [11905.505231] LustreError: 439451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11905.510063] LustreError: 439451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11910.624985] LustreError: 435873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.629687] LustreError: 435873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11915.744619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.745043] LustreError: 439451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.749955] Lustre: Skipped 1 previous similar message [11915.750681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11917.782491] Lustre: server umount lustre-OST0000 complete [11925.369205] Lustre: server umount lustre-MDT0000 complete [11930.211787] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11931.243664] Key type lgssc unregistered [11931.366324] LNet: 440630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11931.368755] LNetError: 440630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11931.379308] LNet: Removed LNI 192.168.201.111@tcp [11931.688515] Key type .llcrypt unregistered [11931.689680] Key type ._llcrypt unregistered [11944.447167] Key type ._llcrypt registered [11944.448357] Key type .llcrypt registered [11944.484694] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11944.598373] Key type .llcrypt unregistered [11944.599327] Key type ._llcrypt unregistered [11951.249277] Key type ._llcrypt registered [11951.250589] Key type .llcrypt registered [11951.298530] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11951.787966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11951.810456] alg: No test for adler32 (adler32-zlib) [11952.660875] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [11952.744626] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11954.319156] Key type lgssc registered [11954.663973] Lustre: Echo OBD driver; http://www.lustre.org/ [11963.459895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11963.464834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.538543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11964.547991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11964.581115] Lustre: lustre-MDT0000: new disk, initializing [11964.601245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11964.606843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11965.920491] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11969.945403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.001312] Lustre: 443984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11970.023942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11970.025734] Lustre: Skipped 1 previous similar message [11970.060808] Lustre: lustre-MDT0001: new disk, initializing [11970.090726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11970.101960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11970.105778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11971.398046] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11974.931489] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11976.685680] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11978.791469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11978.892537] Lustre: lustre-OST0000: new disk, initializing [11978.894608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11978.897935] Lustre: 445949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11978.927181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11980.810594] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11981.247379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11981.251633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11981.286133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11984.682168] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11990.495698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11990.497744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11990.501893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.520702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.522080] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.522933] Lustre: Skipped 1 previous similar message [11991.527138] Lustre: Skipped 1 previous similar message [11993.176072] Lustre: server umount lustre-MDT0000 complete [11994.435457] LustreError: 443974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837696 with bad export cookie 13255950271947833891 [11994.436136] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11994.441446] LustreError: 443974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11994.558825] Lustre: server umount lustre-MDT0001 complete [12006.176436] Lustre: server umount lustre-OST0000 complete [12010.449549] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:48:31 (1786837711) [12013.143164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12013.341332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12014.711456] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12017.798399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12017.930246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12019.332377] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12021.914981] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12023.883986] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12039.203634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12039.322210] Lustre: Found index 0 for lustre-OST0000, updating log [12039.393303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12041.561412] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12044.190686] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12059.877465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12069.344107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12069.346858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.352697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12069.354832] Lustre: Skipped 1 previous similar message [12072.544502] Lustre: server umount lustre-OST0000 complete [12085.353423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12085.357573] Lustre: Skipped 1 previous similar message [12085.359668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12085.362015] Lustre: Skipped 4 previous similar messages [12090.847936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12090.851186] Lustre: Skipped 1 previous similar message [12091.479085] Lustre: server umount lustre-MDT0000 complete [12092.778159] LustreError: 449813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837794 with bad export cookie 13255950271947835270 [12092.781121] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12092.782506] LustreError: 449813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12092.909269] Lustre: server umount lustre-MDT0001 complete [12096.047610] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:49:57 (1786837797) [12101.215429] LNet: 1 local NIs in recovery (showing 1): 192.168.201.111@tcp [12675.983792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12676.126336] LustreError: 454398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12676.133637] LustreError: 454398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12676.148141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12677.342812] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12679.923053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12680.030207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12681.261385] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12684.337650] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12686.535283] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12702.314929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12702.482915] Lustre: Found index 0 for lustre-OST0000, updating log [12702.677633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12704.756052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12706.803459] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12712.203769] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12714.976086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12714.980374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12714.991476] Lustre: Skipped 1 previous similar message [12714.998086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12728.287182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12728.345496] Lustre: server umount lustre-OST0000 complete [12728.931594] LNet: 1 local NIs in recovery (showing 1): 192.168.201.111@tcp [12730.976983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12730.978447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.981212] Lustre: Skipped 2 previous similar messages [12730.982912] Lustre: Skipped 1 previous similar message [12733.024212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12733.026693] Lustre: Skipped 4 previous similar messages [12736.098868] LustreError: 454398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12736.107899] LustreError: 454398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12736.139582] Lustre: server umount lustre-MDT0000 complete [12737.120382] LustreError: 457003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12737.126600] LustreError: 457003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12737.832718] LustreError: 454379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838439 with bad export cookie 13255950271947836341 [12737.835074] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12737.839872] LustreError: 454379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12737.999499] Lustre: server umount lustre-MDT0001 complete [12757.172781] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [12761.639497] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12777.910975] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:01:19 (1786838479) [12785.108024] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12786.252699] Key type lgssc unregistered [12786.391410] LNet: 461020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12786.394918] LNetError: 461020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12834.533336] LNet: Removed LNI 192.168.201.111@tcp [12834.935497] Key type .llcrypt unregistered [12834.939249] Key type ._llcrypt unregistered [12843.107203] Key type ._llcrypt registered [12843.108326] Key type .llcrypt registered [12843.168234] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12843.635899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12843.681782] alg: No test for adler32 (adler32-zlib) [12844.564089] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [12844.683585] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12846.271178] Key type lgssc registered [12846.770096] Lustre: Echo OBD driver; http://www.lustre.org/ [12848.816047] LNet: Added LNI 192.168.201.111@tcp100 [8/256/0/180] [12849.237102] LNet: Added LNI 192.168.201.111@tcp101 [8/256/0/180] [12849.646373] LNet: Added LNI 192.168.201.111@tcp102 [8/256/0/180] [12850.068814] LNet: Added LNI 192.168.201.111@tcp103 [8/256/0/180] [12850.517754] LNet: Added LNI 192.168.201.111@tcp104 [8/256/0/180] [12850.924880] LNet: Added LNI 192.168.201.111@tcp105 [8/256/0/180] [12851.313633] LNet: Added LNI 192.168.201.111@tcp106 [8/256/0/180] [12851.749594] LNet: Added LNI 192.168.201.111@tcp107 [8/256/0/180] [12852.198244] LNet: Added LNI 192.168.201.111@tcp108 [8/256/0/180] [12852.622263] LNet: Added LNI 192.168.201.111@tcp109 [8/256/0/180] [12853.008705] LNet: Added LNI 192.168.201.111@tcp110 [8/256/0/180] [12853.425732] LNet: Added LNI 192.168.201.111@tcp111 [8/256/0/180] [12853.871261] LNet: Added LNI 192.168.201.111@tcp112 [8/256/0/180] [12854.244223] LNet: Added LNI 192.168.201.111@tcp113 [8/256/0/180] [12854.671323] LNet: Added LNI 192.168.201.111@tcp114 [8/256/0/180] [12855.113495] LNet: Added LNI 192.168.201.111@tcp115 [8/256/0/180] [12855.543480] LNet: Added LNI 192.168.201.111@tcp116 [8/256/0/180] [12855.998036] LNet: Added LNI 192.168.201.111@tcp117 [8/256/0/180] [12856.418308] LNet: Added LNI 192.168.201.111@tcp118 [8/256/0/180] [12856.805607] LNet: Added LNI 192.168.201.111@tcp119 [8/256/0/180] [12857.217347] LNet: Added LNI 192.168.201.111@tcp120 [8/256/0/180] [12857.634717] LNet: Added LNI 192.168.201.111@tcp121 [8/256/0/180] [12858.104330] LNet: Added LNI 192.168.201.111@tcp122 [8/256/0/180] [12858.532284] LNet: Added LNI 192.168.201.111@tcp123 [8/256/0/180] [12858.983759] LNet: Added LNI 192.168.201.111@tcp124 [8/256/0/180] [12859.387917] LNet: Added LNI 192.168.201.111@tcp125 [8/256/0/180] [12859.793852] LNet: Added LNI 192.168.201.111@tcp126 [8/256/0/180] [12860.188371] LNet: Added LNI 192.168.201.111@tcp127 [8/256/0/180] [12860.566432] LNet: Added LNI 192.168.201.111@tcp128 [8/256/0/180] [12860.992345] LNet: Added LNI 192.168.201.111@tcp129 [8/256/0/180] [12861.348777] LNet: Added LNI 192.168.201.111@tcp130 [8/256/0/180] [12861.741949] LNet: Added LNI 192.168.201.111@tcp131 [8/256/0/180] [12862.132300] LNet: Added LNI 192.168.201.111@tcp132 [8/256/0/180] [12862.541852] LNet: Added LNI 192.168.201.111@tcp133 [8/256/0/180] [12862.970189] LNet: Added LNI 192.168.201.111@tcp134 [8/256/0/180] [12863.373511] LNet: Added LNI 192.168.201.111@tcp135 [8/256/0/180] [12866.190866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12866.202444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.248981] Lustre: 464015:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [12867.283303] LNetError: 461478:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [12867.287253] LNetError: Protocol error connecting to 192.168.201.111@tcp117 at host 192.168.201.111:988: is it running a compatible version of Lustre? [12867.344336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12867.361354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12867.400749] Lustre: lustre-MDT0000: new disk, initializing [12867.426651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12867.435117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12868.321844] LNetError: 461478:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [12868.326101] LNetError: Protocol error connecting to 192.168.201.111@tcp102 at host 192.168.201.111:988: is it running a compatible version of Lustre? [12869.261714] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12874.767721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12874.853367] Lustre: 464033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12874.888512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12874.891398] Lustre: Skipped 1 previous similar message [12874.932938] Lustre: lustre-MDT0001: new disk, initializing [12874.950364] Lustre: 464835:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [12874.954331] Lustre: 464835:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12875.010382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12875.034148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12875.038619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12877.000174] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12882.252566] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12882.443764] Lustre: 465452:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [12882.452625] Lustre: 465452:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12885.020502] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12888.379109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12888.566419] Lustre: lustre-OST0000: new disk, initializing [12888.568181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12888.570722] Lustre: 466004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12888.622029] Lustre: 466009:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.111@tcp: can't add NID 192.168.201.111@tcp130: rc = -75 [12888.624792] Lustre: 466009:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12888.649396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12891.737270] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12892.396848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12892.407173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12892.473327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12896.878205] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12902.883548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12902.884591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12902.883924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12902.891840] Lustre: Skipped 1 previous similar message [12902.898621] Lustre: Skipped 1 previous similar message [12905.120676] LNetError: 461479:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [12905.124090] LNetError: Protocol error connecting to 192.168.201.111@tcp117 at host 192.168.201.111:988: is it running a compatible version of Lustre? [12908.003285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12908.005773] Lustre: Skipped 1 previous similar message [12908.192227] LNetError: 464043:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [12908.195468] LNetError: Protocol error connecting to 192.168.201.111@tcp102 at host 192.168.201.111:988: is it running a compatible version of Lustre? [12911.013229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12911.016797] Lustre: Skipped 1 previous similar message [12914.144816] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12914.225545] Lustre: server umount lustre-OST0000 complete [12916.193156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12916.193937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.196337] LustreError: Skipped 1 previous similar message [12916.197138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12916.201136] Lustre: Skipped 1 previous similar message [12917.408402] LNetError: 464035:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [12917.412384] LNetError: Protocol error connecting to 192.168.201.111@tcp117 at host 192.168.201.111:988: is it running a compatible version of Lustre? [12921.312237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12921.314779] Lustre: Skipped 2 previous similar messages [12930.015154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12930.087615] Lustre: server umount lustre-MDT0000 complete [12930.723390] LNetError: 464036:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.111 [12930.727460] LNetError: 464036:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [12930.730304] LNetError: Protocol error connecting to 192.168.201.111@tcp117 at host 192.168.201.111:988: is it running a compatible version of Lustre? [12930.736055] LNetError: Skipped 1 previous similar message [12931.590680] LustreError: 464023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838633 with bad export cookie 12655942073425020909 [12931.591239] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12931.596847] LustreError: 464023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12931.734713] Lustre: server umount lustre-MDT0001 complete [12936.168863] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12937.329368] Key type lgssc unregistered [12937.513720] LNet: 467919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12937.517281] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12937.520236] LNet: Removed LNI 192.168.201.111@tcp [12938.527166] LNet: Removed LNI 192.168.201.111@tcp100 [12938.528527] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12938.530579] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12938.532694] LNet: Removed LNI 192.168.201.111@tcp101 [12939.551132] LNet: Removed LNI 192.168.201.111@tcp102 [12939.552719] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12939.555042] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12939.557535] LNet: Removed LNI 192.168.201.111@tcp103 [12940.575099] LNet: Removed LNI 192.168.201.111@tcp104 [12941.599144] LNet: Removed LNI 192.168.201.111@tcp105 [12941.601411] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12941.604631] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12941.607450] LNet: Removed LNI 192.168.201.111@tcp106 [12942.623201] LNet: Removed LNI 192.168.201.111@tcp107 [12943.647150] LNet: Removed LNI 192.168.201.111@tcp108 [12944.671139] LNet: Removed LNI 192.168.201.111@tcp109 [12945.695222] LNet: Removed LNI 192.168.201.111@tcp110 [12945.696907] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12945.700655] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12945.703690] LNet: Removed LNI 192.168.201.111@tcp111 [12946.719226] LNet: Removed LNI 192.168.201.111@tcp112 [12947.743132] LNet: Removed LNI 192.168.201.111@tcp113 [12948.767158] LNet: Removed LNI 192.168.201.111@tcp114 [12949.791141] LNet: Removed LNI 192.168.201.111@tcp115 [12950.815276] LNet: Removed LNI 192.168.201.111@tcp116 [12951.840415] LNet: Removed LNI 192.168.201.111@tcp117 [12952.863151] LNet: Removed LNI 192.168.201.111@tcp118 [12953.887158] LNet: Removed LNI 192.168.201.111@tcp119 [12953.888556] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12953.890515] LNetError: 467919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12953.892630] LNet: Removed LNI 192.168.201.111@tcp120 [12954.911143] LNet: Removed LNI 192.168.201.111@tcp121 [12954.912791] LNet: Removed LNI 192.168.201.111@tcp122 [12954.914146] LNet: Removed LNI 192.168.201.111@tcp123 [12954.915896] LNet: Removed LNI 192.168.201.111@tcp124 [12954.917122] LNet: Removed LNI 192.168.201.111@tcp125 [12954.918587] LNet: Removed LNI 192.168.201.111@tcp126 [12954.920295] LNet: Removed LNI 192.168.201.111@tcp127 [12954.921888] LNet: Removed LNI 192.168.201.111@tcp128 [12954.922951] LNet: Removed LNI 192.168.201.111@tcp129 [12954.924400] LNet: Removed LNI 192.168.201.111@tcp130 [12954.925709] LNet: Removed LNI 192.168.201.111@tcp131 [12954.926884] LNet: Removed LNI 192.168.201.111@tcp132 [12954.928414] LNet: Removed LNI 192.168.201.111@tcp133 [12954.929762] LNet: Removed LNI 192.168.201.111@tcp134 [12954.936527] LNet: Removed LNI 192.168.201.111@tcp135 [12955.224838] Key type .llcrypt unregistered [12955.226345] Key type ._llcrypt unregistered [12963.937276] Key type ._llcrypt registered [12963.938647] Key type .llcrypt registered [12963.992312] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [12970.593531] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12971.011837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12971.073143] alg: No test for adler32 (adler32-zlib) [12971.947726] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [12972.048037] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12973.639165] Key type lgssc registered [12974.077942] Lustre: Echo OBD driver; http://www.lustre.org/ [12987.987921] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:04:48 (1786838688) [12998.301968] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12999.707187] Key type lgssc unregistered [12999.882845] LNet: 471982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12999.888972] LNetError: 471982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12999.900467] LNet: Removed LNI 192.168.201.111@tcp [13000.289191] Key type .llcrypt unregistered [13000.290370] Key type ._llcrypt unregistered [13010.205479] Key type ._llcrypt registered [13010.207882] Key type .llcrypt registered [13010.277156] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13010.892765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13010.931672] alg: No test for adler32 (adler32-zlib) [13011.834935] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [13011.981484] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [13013.591148] Key type lgssc registered [13014.109173] Lustre: Echo OBD driver; http://www.lustre.org/ [13017.969076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13017.979586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13019.083286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13019.103267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13019.142826] Lustre: lustre-MDT0000: new disk, initializing [13019.186446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13019.197339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13020.935320] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13025.918685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.971239] Lustre: 473172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13026.003491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13026.005968] Lustre: Skipped 1 previous similar message [13026.039264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13026.049805] Lustre: lustre-MDT0001: new disk, initializing [13026.087728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13026.101733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13026.106346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13027.669269] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13032.040480] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13034.348548] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13037.001363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13037.115973] Lustre: lustre-OST0000: new disk, initializing [13037.117564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13037.119788] Lustre: 475139:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13037.152877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13039.475239] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13042.170838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13042.174945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13042.218291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13043.539536] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13044.929632] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing lnet_if_list [13046.772881] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13046.787287] LNet: Added LNI 192.168.201.111@tcp10 [8/256/0/180] [13048.861104] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13048.874485] LNet: Added LNI 192.168.201.111@tcp11 [8/256/0/180] [13050.350205] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13050.363653] LNet: Added LNI 192.168.201.111@tcp12 [8/256/0/180] [13051.773178] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13051.784585] LNet: Added LNI 192.168.201.111@tcp13 [8/256/0/180] [13053.263773] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13053.277174] LNet: Added LNI 192.168.201.111@tcp14 [8/256/0/180] [13054.819413] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13054.833243] LNet: Added LNI 192.168.201.111@tcp15 [8/256/0/180] [13056.616869] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13056.631747] LNet: Added LNI 192.168.201.111@tcp16 [8/256/0/180] [13058.418213] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13058.432079] LNet: Added LNI 192.168.201.111@tcp17 [8/256/0/180] [13060.204227] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13060.218727] LNet: Added LNI 192.168.201.111@tcp18 [8/256/0/180] [13062.187618] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13062.200695] LNet: Added LNI 192.168.201.111@tcp19 [8/256/0/180] [13064.090922] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13064.104681] LNet: Added LNI 192.168.201.111@tcp20 [8/256/0/180] [13065.901102] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13065.915938] LNet: Added LNI 192.168.201.111@tcp21 [8/256/0/180] [13067.740867] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13067.754161] LNet: Added LNI 192.168.201.111@tcp22 [8/256/0/180] [13069.691201] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13069.702236] LNet: Added LNI 192.168.201.111@tcp23 [8/256/0/180] [13071.605163] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13071.620856] LNet: Added LNI 192.168.201.111@tcp24 [8/256/0/180] [13073.623900] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13073.641552] LNet: Added LNI 192.168.201.111@tcp25 [8/256/0/180] [13075.814732] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13075.829235] LNet: Added LNI 192.168.201.111@tcp26 [8/256/0/180] [13077.980932] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13077.997321] LNet: Added LNI 192.168.201.111@tcp27 [8/256/0/180] [13080.066961] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13080.079942] LNet: Added LNI 192.168.201.111@tcp28 [8/256/0/180] [13082.296062] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13082.307815] LNet: Added LNI 192.168.201.111@tcp29 [8/256/0/180] [13084.995803] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13085.010865] LNet: Added LNI 192.168.201.111@tcp30 [8/256/0/180] [13087.499288] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13087.514595] LNet: Added LNI 192.168.201.111@tcp31 [8/256/0/180] [13090.086064] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13090.099048] LNet: Added LNI 192.168.201.111@tcp32 [8/256/0/180] [13092.506922] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13092.518947] LNet: Added LNI 192.168.201.111@tcp33 [8/256/0/180] [13094.899287] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13094.913126] LNet: Added LNI 192.168.201.111@tcp34 [8/256/0/180] [13097.417173] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13097.430324] LNet: Added LNI 192.168.201.111@tcp35 [8/256/0/180] [13100.242747] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13100.255816] LNet: Added LNI 192.168.201.111@tcp36 [8/256/0/180] [13102.829148] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13102.841134] LNet: Added LNI 192.168.201.111@tcp37 [8/256/0/180] [13105.313559] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13105.329058] LNet: Added LNI 192.168.201.111@tcp38 [8/256/0/180] [13108.235122] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13108.248958] LNet: Added LNI 192.168.201.111@tcp39 [8/256/0/180] [13111.568968] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13111.592367] LNet: Added LNI 192.168.201.111@tcp40 [8/256/0/180] [13114.984621] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13115.000215] LNet: Added LNI 192.168.201.111@tcp41 [8/256/0/180] [13118.396566] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13118.420073] LNet: Added LNI 192.168.201.111@tcp42 [8/256/0/180] [13121.734168] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13121.748164] LNet: Added LNI 192.168.201.111@tcp43 [8/256/0/180] [13124.700409] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13124.715744] LNet: Added LNI 192.168.201.111@tcp44 [8/256/0/180] [13127.881944] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13127.896949] LNet: Added LNI 192.168.201.111@tcp45 [8/256/0/180] [13131.043524] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13131.059146] LNet: Added LNI 192.168.201.111@tcp46 [8/256/0/180] [13133.894332] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13133.908762] LNet: Added LNI 192.168.201.111@tcp47 [8/256/0/180] [13137.411873] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13137.435267] LNet: Added LNI 192.168.201.111@tcp48 [8/256/0/180] [13140.972379] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13140.987868] LNet: Added LNI 192.168.201.111@tcp49 [8/256/0/180] [13144.418506] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13144.435315] LNet: Added LNI 192.168.201.111@tcp50 [8/256/0/180] [13154.271780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13154.273840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.278924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13154.785554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.790898] Lustre: Skipped 1 previous similar message [13159.644432] Lustre: server umount lustre-MDT0000 complete [13160.955124] LustreError: 473164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838863 with bad export cookie 5958101240240730323 [13160.959174] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13160.959348] LustreError: 473164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13161.074481] Lustre: server umount lustre-MDT0001 complete [13169.018683] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13172.149669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13181.407130] Lustre: 472447:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838867/real 1786838867] req@00000000f2ab31c3 x1873636191460224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786838883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13181.419187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13186.527127] Lustre: 472444:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838872/real 1786838872] req@00000000c2c60902 x1873636191460352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786838888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13186.527321] LustreError: 472443:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000071fda8e9 x1873636191461248/t0(0) o250->MGC192.168.201.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13186.628122] LustreError: 475132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.638684] LustreError: 475132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13186.645122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13187.679108] Lustre: 472444:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838873/real 1786838873] req@000000009f91e5ed x1873636191460608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786838889 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13188.072744] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13190.688215] LustreError: 475133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13191.167448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13191.303641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13192.287135] Lustre: 472446:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838878/real 1786838878] req@0000000069e6cb11 x1873636191460736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786838894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13192.356400] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13192.736420] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13195.464518] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13197.566205] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13197.791119] Lustre: 472446:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838883/real 1786838883] req@000000009271b6e2 x1873636191460864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786838899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13207.009426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13207.012211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13207.016465] LustreError: Skipped 1 previous similar message [13207.017327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13207.017330] Lustre: Skipped 2 previous similar messages [13207.029705] Lustre: Skipped 1 previous similar message [13212.193335] Lustre: server umount lustre-OST0000 complete [13217.248191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13217.252290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13217.257916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13217.260525] Lustre: Skipped 3 previous similar messages [13219.744733] Lustre: server umount lustre-MDT0000 complete [13221.242181] LustreError: 473165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838923 with bad export cookie 5958101240240732479 [13221.245135] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13221.245810] LustreError: 473165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13221.364336] Lustre: server umount lustre-MDT0001 complete [13225.317652] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13226.446237] Key type lgssc unregistered [13226.608556] LNet: 502105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13226.612027] LNetError: 502105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13226.615189] LNet: Removed LNI 192.168.201.111@tcp [13226.616926] LNet: Removed LNI 192.168.201.111@tcp10 [13226.618615] LNet: Removed LNI 192.168.201.111@tcp11 [13226.620352] LNet: Removed LNI 192.168.201.111@tcp12 [13226.622140] LNet: Removed LNI 192.168.201.111@tcp13 [13226.624111] LNet: Removed LNI 192.168.201.111@tcp14 [13226.625893] LNet: Removed LNI 192.168.201.111@tcp15 [13226.628072] LNet: Removed LNI 192.168.201.111@tcp16 [13226.630171] LNet: Removed LNI 192.168.201.111@tcp17 [13226.631990] LNet: Removed LNI 192.168.201.111@tcp18 [13226.633743] LNet: Removed LNI 192.168.201.111@tcp19 [13226.635432] LNet: Removed LNI 192.168.201.111@tcp20 [13226.637247] LNet: Removed LNI 192.168.201.111@tcp21 [13226.639140] LNet: Removed LNI 192.168.201.111@tcp22 [13226.640985] LNet: Removed LNI 192.168.201.111@tcp23 [13226.642838] LNet: Removed LNI 192.168.201.111@tcp24 [13226.644669] LNet: Removed LNI 192.168.201.111@tcp25 [13226.646311] LNet: Removed LNI 192.168.201.111@tcp26 [13226.648276] LNet: Removed LNI 192.168.201.111@tcp27 [13226.650087] LNet: Removed LNI 192.168.201.111@tcp28 [13226.651727] LNet: Removed LNI 192.168.201.111@tcp29 [13226.653465] LNet: Removed LNI 192.168.201.111@tcp30 [13226.655467] LNet: Removed LNI 192.168.201.111@tcp31 [13226.657176] LNet: Removed LNI 192.168.201.111@tcp32 [13226.658771] LNet: Removed LNI 192.168.201.111@tcp33 [13226.660237] LNet: Removed LNI 192.168.201.111@tcp34 [13226.661710] LNet: Removed LNI 192.168.201.111@tcp35 [13226.663284] LNet: Removed LNI 192.168.201.111@tcp36 [13226.664659] LNet: Removed LNI 192.168.201.111@tcp37 [13226.666214] LNet: Removed LNI 192.168.201.111@tcp38 [13226.667739] LNet: Removed LNI 192.168.201.111@tcp39 [13226.668931] LNet: Removed LNI 192.168.201.111@tcp40 [13226.669993] LNet: Removed LNI 192.168.201.111@tcp41 [13226.671454] LNet: Removed LNI 192.168.201.111@tcp42 [13226.672891] LNet: Removed LNI 192.168.201.111@tcp43 [13226.674291] LNet: Removed LNI 192.168.201.111@tcp44 [13226.675983] LNet: Removed LNI 192.168.201.111@tcp45 [13226.677480] LNet: Removed LNI 192.168.201.111@tcp46 [13226.679046] LNet: Removed LNI 192.168.201.111@tcp47 [13226.680297] LNet: Removed LNI 192.168.201.111@tcp48 [13226.681707] LNet: Removed LNI 192.168.201.111@tcp49 [13226.688915] LNet: Removed LNI 192.168.201.111@tcp50 [13226.970519] Key type .llcrypt unregistered [13226.971888] Key type ._llcrypt unregistered [13234.133624] Key type ._llcrypt registered [13234.134808] Key type .llcrypt registered [13234.175726] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13240.935317] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13241.367951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13241.406535] alg: No test for adler32 (adler32-zlib) [13242.272960] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [13242.375700] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [13243.959161] Key type lgssc registered [13244.373197] Lustre: Echo OBD driver; http://www.lustre.org/ [13256.288748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13256.294280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13257.376827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13257.388035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13257.423502] Lustre: lustre-MDT0000: new disk, initializing [13257.447200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13257.454119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13258.828145] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13263.068379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13263.109641] Lustre: 505275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13263.131693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13263.133986] Lustre: Skipped 1 previous similar message [13263.170147] Lustre: lustre-MDT0001: new disk, initializing [13263.201138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13263.213291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13263.216546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13264.556429] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13268.501781] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13270.420822] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13273.148209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13273.234715] Lustre: *** cfs_fail_loc=90f, val=0*** [13273.265494] Lustre: lustre-OST0000: new disk, initializing [13273.267265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13273.270177] Lustre: 507292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13273.301508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13275.349770] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13278.718608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13278.725198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13278.760734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13279.376552] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13283.811072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13283.811109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13283.813929] LustreError: Skipped 1 previous similar message [13283.818481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13283.821739] Lustre: Skipped 1 previous similar message [13288.148164] Lustre: server umount lustre-OST0000 complete [13294.049327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13294.050464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13294.056923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13294.060789] Lustre: Skipped 2 previous similar messages [13295.591945] Lustre: server umount lustre-MDT0000 complete [13296.949362] LustreError: 506072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838999 with bad export cookie 10046145695929438433 [13296.952382] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13296.953892] LustreError: 506072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13297.072183] Lustre: server umount lustre-MDT0001 complete [13300.970148] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13302.006269] Key type lgssc unregistered [13302.130321] LNet: 509198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13302.132714] LNetError: 509198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13302.140272] LNet: Removed LNI 192.168.201.111@tcp [13302.447726] Key type .llcrypt unregistered [13302.449047] Key type ._llcrypt unregistered [13310.751532] Key type ._llcrypt registered [13310.752588] Key type .llcrypt registered [13310.792823] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13311.226597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13311.236656] alg: No test for adler32 (adler32-zlib) [13312.108090] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [13312.212742] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [13313.799178] Key type lgssc registered [13314.241340] Lustre: Echo OBD driver; http://www.lustre.org/ [13317.380783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13317.396492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13318.626487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13320.464447] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13324.215991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13324.391204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13326.211979] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13329.364493] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13331.777021] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13334.547776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13334.625485] LustreError: 512263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13334.631402] LustreError: 512263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13334.719603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13337.174799] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13340.469636] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13349.925272] loop0: detected capacity change from 0 to 204800000 [13350.034652] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13350.038263] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13350.046492] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13352.252221] loop0: detected capacity change from 0 to 204800000 [13352.380233] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13352.384880] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13352.390375] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13354.099702] loop0: detected capacity change from 0 to 204800000 [13355.922100] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13355.970883] Lustre: fs15246-OST0000: new disk, initializing [13355.972651] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13355.974883] Lustre: 514035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13355.993325] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13358.154887] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13362.342299] loop1: detected capacity change from 0 to 204800000 [13364.316264] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13364.365862] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13364.393522] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13364.429519] Lustre: fs15246-MDT0000: new disk, initializing [13364.477258] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13364.487454] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13364.530241] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13365.535836] Lustre: 515156:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000003f29d817 x1873636505976704/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786839083 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13365.904599] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13374.311083] LustreError: 515156:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [13374.315649] LustreError: 515156:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [13374.320217] LustreError: 515156:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [13377.515420] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13377.523967] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13380.432933] Lustre: server umount fs15246-MDT0000 complete [13382.628237] LustreError: 513002:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13384.005079] Lustre: server umount fs15246-OST0000 complete [13389.280051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13389.282355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13389.286082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13391.328085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13391.331099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13391.336879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13394.001488] Lustre: server umount lustre-OST0000 complete [13395.935823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13395.938482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13395.943966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13395.946406] Lustre: Skipped 2 previous similar messages [13401.563951] Lustre: server umount lustre-MDT0000 complete [13402.998201] LustreError: 510470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839105 with bad export cookie 18255669465054434240 [13403.001682] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13403.002556] LustreError: 510470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13403.130467] Lustre: server umount lustre-MDT0001 complete [13407.255563] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13408.356061] Key type lgssc unregistered [13408.492400] LNet: 518250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13408.495061] LNetError: 518250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13408.504276] LNet: Removed LNI 192.168.201.111@tcp [13408.837894] Key type .llcrypt unregistered [13408.839260] Key type ._llcrypt unregistered [13416.537826] Key type ._llcrypt registered [13416.539460] Key type .llcrypt registered [13416.581608] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13417.034777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13417.061233] alg: No test for adler32 (adler32-zlib) [13417.912030] Lustre: Lustre: Build Version: 2.17.57_5_g4421514 [13417.998601] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [13419.575152] Key type lgssc registered [13419.934722] Lustre: Echo OBD driver; http://www.lustre.org/ [13432.960143] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13436.006319] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13449.036050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13449.044230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13450.139237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13450.150823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13450.190228] Lustre: lustre-MDT0000: new disk, initializing [13450.217844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13450.225140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13451.742974] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13456.382445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13456.429497] Lustre: 522686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13456.454588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13456.456992] Lustre: Skipped 1 previous similar message [13456.494743] Lustre: lustre-MDT0001: new disk, initializing [13456.540500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13456.554495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13456.558325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13458.101650] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13462.384502] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13464.474071] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13466.965518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13467.111936] Lustre: lustre-OST0000: new disk, initializing [13467.114962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13467.118698] Lustre: 524652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13467.154756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13469.522606] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13473.719945] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13475.405715] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13476.854870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13476.858041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13476.888143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13477.523772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [13478.966204] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13479.044550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13481.954484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13481.954972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13481.956069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13481.957041] LustreError: Skipped 1 previous similar message [13481.961052] Lustre: Skipped 1 previous similar message [13485.839321] Lustre: server umount lustre-OST0000 complete [13487.072074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13487.072609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13487.075489] Lustre: Skipped 1 previous similar message [13487.078735] Lustre: Skipped 2 previous similar messages [13492.192131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13492.194098] Lustre: Skipped 1 previous similar message [13493.209590] Lustre: server umount lustre-MDT0000 complete [13494.657353] LustreError: 526207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839196 with bad export cookie 16031792962134754440 [13494.660157] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13494.660859] LustreError: 526207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13494.777936] Lustre: server umount lustre-MDT0001 complete [13503.205122] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13506.763366] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13517.469364] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13237 sec ========= 20:13:38 (1786839218) [13518.087383] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:13:39 (1786839219) === [13519.375890] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:13:40 (1786839220) === [13533.412735] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13534.485679] Key type lgssc unregistered [13534.620298] LNet: 531301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13534.622728] LNetError: 531301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13534.634304] LNet: Removed LNI 192.168.201.111@tcp [13534.961698] Key type .llcrypt unregistered [13534.962846] Key type ._llcrypt unregistered