[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468563888 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003108] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008309] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011019] pid_max: default: 32768 minimum: 301 [ 0.012131] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.015066] *** VALIDATE selinux *** [ 0.023854] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028442] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031025] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] *** VALIDATE tmpfs *** [ 0.033462] *** VALIDATE proc *** [ 0.034238] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036222] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039045] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043273] debug: unmapping init [mem 0xffffffff98e59000-0xffffffff98e60fff] [ 0.045175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049018] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054294] rcu: Hierarchical SRCU implementation. [ 0.056414] smp: Bringing up secondary CPUs ... [ 0.057639] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061259] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146517] node 0 deferred pages initialised in 80ms [ 0.148152] devtmpfs: initialized [ 0.149182] x86/mm: Memory block size: 128MB [ 0.152842] gcov: version magic: 0x41383552 [ 0.155352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157303] pinctrl core: initialized pinctrl subsystem [ 0.158169] [ 0.158586] ************************************************************* [ 0.159011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160011] ** ** [ 0.161010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162007] ** ** [ 0.163009] ** This means that this kernel is built to expose internal ** [ 0.164014] ** IOMMU data structures, which may compromise security on ** [ 0.165015] ** your system. ** [ 0.166011] ** ** [ 0.167011] ** If you see this message and you are not debugging the ** [ 0.168014] ** kernel, report this immediately to your vendor! ** [ 0.169014] ** ** [ 0.170015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171010] ************************************************************* [ 0.172728] NET: Registered protocol family 16 [ 0.173339] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176446] cpuidle: using governor menu [ 0.178796] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182485] PCI: Using configuration type 1 for base access [ 0.184122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194176] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.201140] cryptd: max_cpu_qlen set to 1000 [ 0.203309] ACPI: Added _OSI(Module Device) [ 0.204016] ACPI: Added _OSI(Processor Device) [ 0.205029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211344] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218721] ACPI: Interpreter enabled [ 0.220056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221010] ACPI: Using IOAPIC for interrupt routing [ 0.222125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225302] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.235684] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249618] acpiphp: Slot [2] registered [ 0.251128] acpiphp: Slot [5] registered [ 0.252141] acpiphp: Slot [6] registered [ 0.254157] acpiphp: Slot [7] registered [ 0.256119] acpiphp: Slot [8] registered [ 0.257104] acpiphp: Slot [9] registered [ 0.259126] acpiphp: Slot [10] registered [ 0.260121] acpiphp: Slot [3] registered [ 0.262122] acpiphp: Slot [4] registered [ 0.264124] acpiphp: Slot [11] registered [ 0.265112] acpiphp: Slot [12] registered [ 0.267181] acpiphp: Slot [13] registered [ 0.268073] acpiphp: Slot [14] registered [ 0.269104] acpiphp: Slot [15] registered [ 0.271110] acpiphp: Slot [16] registered [ 0.272107] acpiphp: Slot [17] registered [ 0.274117] acpiphp: Slot [18] registered [ 0.275093] acpiphp: Slot [19] registered [ 0.276081] acpiphp: Slot [20] registered [ 0.277078] acpiphp: Slot [21] registered [ 0.279115] acpiphp: Slot [22] registered [ 0.280107] acpiphp: Slot [23] registered [ 0.281098] acpiphp: Slot [24] registered [ 0.283169] acpiphp: Slot [25] registered [ 0.284098] acpiphp: Slot [26] registered [ 0.286138] acpiphp: Slot [27] registered [ 0.287114] acpiphp: Slot [28] registered [ 0.289137] acpiphp: Slot [29] registered [ 0.291086] acpiphp: Slot [30] registered [ 0.292120] acpiphp: Slot [31] registered [ 0.294086] PCI host bridge to bus 0000:00 [ 0.296019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314195] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318332] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327508] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.332361] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.334018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339055] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345704] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348929] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353812] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385960] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.426026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.437901] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.446014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494826] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.502015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.509015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.545368] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.553015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.560018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.589473] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.597051] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.605015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645010] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.661014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.688017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.699184] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.702375] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.705370] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.708411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.710229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.717128] iommu: Default domain type: Passthrough [ 0.719521] SCSI subsystem initialized [ 0.721215] ACPI: bus type USB registered [ 0.723160] usbcore: registered new interface driver usbfs [ 0.726118] usbcore: registered new interface driver hub [ 0.727128] usbcore: registered new device driver usb [ 0.729206] pps_core: LinuxPPS API ver. 1 registered [ 0.731015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.734087] PTP clock support registered [ 0.736158] EDAC MC: Ver: 3.0.0 [ 0.738224] PCI: Using ACPI for IRQ routing [ 0.739838] NetLabel: Initializing [ 0.740009] NetLabel: domain hash size = 128 [ 0.741008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.742207] NetLabel: unlabeled traffic allowed by default [ 0.744144] vgaarb: loaded [ 0.746303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.748016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.756321] clocksource: Switched to clocksource kvm-clock [ 0.861786] VFS: Disk quotas dquot_6.6.0 [ 0.863585] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.865805] *** VALIDATE ramfs *** [ 0.867139] *** VALIDATE hugetlbfs *** [ 0.868690] pnp: PnP ACPI init [ 0.871151] pnp: PnP ACPI: found 6 devices [ 0.892528] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896960] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899365] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901276] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.904442] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906959] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909748] NET: Registered protocol family 2 [ 0.912343] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.918073] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.922753] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.928213] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931572] TCP: Hash tables configured (established 65536 bind 65536) [ 0.934698] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938567] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941496] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944932] NET: Registered protocol family 1 [ 0.947909] RPC: Registered named UNIX socket transport module. [ 0.950570] RPC: Registered udp transport module. [ 0.952645] RPC: Registered tcp transport module. [ 0.954705] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.957747] NET: Registered protocol family 44 [ 0.959792] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.962209] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.964216] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.966550] PCI: CLS 0 bytes, default 64 [ 0.968603] Unpacking initramfs... [ 2.466354] debug: unmapping init [mem 0xffff8f80fcc54000-0xffff8f80fffbffff] [ 2.473673] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.475792] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.478465] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.988450] Initialise system trusted keyrings [ 2.990524] Key type blacklist registered [ 2.992415] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.001240] zbud: loaded [ 3.004544] *** VALIDATE nfs *** [ 3.005776] *** VALIDATE nfs4 *** [ 3.007573] pstore: using deflate compression [ 3.011288] Platform Keyring initialized [ 3.109749] NET: Registered protocol family 38 [ 3.112289] Key type asymmetric registered [ 3.114412] Asymmetric key parser 'x509' registered [ 3.117023] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.123550] io scheduler mq-deadline registered [ 3.125373] io scheduler kyber registered [ 3.127146] io scheduler bfq registered [ 3.129456] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.133097] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.136643] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.139677] ACPI: Power Button [PWRF] [ 3.145639] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.153785] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.176213] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.185523] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.209255] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.237124] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.265312] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.270913] Non-volatile memory driver v1.3 [ 3.273108] Linux agpgart interface v0.103 [ 3.312900] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.316329] vda: detected capacity change from 0 to 74645504 [ 3.331341] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.334049] vdb: detected capacity change from 0 to 1073741824 [ 3.351341] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.354615] vdc: detected capacity change from 0 to 2621440000 [ 3.369964] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373068] vdd: detected capacity change from 0 to 2621440000 [ 3.389127] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392058] vde: detected capacity change from 0 to 4294967296 [ 3.409029] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.411823] vdf: detected capacity change from 0 to 4294967296 [ 3.419730] libphy: Fixed MDIO Bus: probed [ 3.425761] usbcore: registered new interface driver usbserial_generic [ 3.428335] usbserial: USB Serial support registered for generic [ 3.432130] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.437809] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.440477] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.443505] mousedev: PS/2 mouse device common for all mice [ 3.447381] rtc_cmos 00:05: RTC can wake from S4 [ 3.450323] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.451267] rtc_cmos 00:05: registered as rtc0 [ 3.457441] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.458899] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.461900] intel_pstate: CPU model not supported [ 3.470131] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.471823] hid: raw HID events driver (C) Jiri Kosina [ 3.476644] usbcore: registered new interface driver usbhid [ 3.478944] usbhid: USB HID core driver [ 3.480857] drop_monitor: Initializing network drop monitor service [ 3.484157] Initializing XFRM netlink socket [ 3.486532] NET: Registered protocol family 10 [ 3.491229] Segment Routing with IPv6 [ 3.492771] NET: Registered protocol family 17 [ 3.495439] mpls_gso: MPLS GSO support [ 3.501298] RAS: Correctable Errors collector initialized. [ 3.503745] AVX version of gcm_enc/dec engaged. [ 3.505418] AES CTR mode by8 optimization enabled [ 3.593380] sched_clock: Marking stable (3593354184, 0)->(4565545149, -972190965) [ 3.598077] registered taskstats version 1 [ 3.600757] Loading compiled-in X.509 certificates [ 3.602876] zswap: loaded using pool lzo/zbud [ 3.629282] Key type big_key registered [ 3.643764] Key type encrypted registered [ 3.648256] ima: No TPM chip found, activating TPM-bypass! [ 3.650368] ima: Allocated hash algorithm: sha1 [ 3.652651] ima: No architecture policies found [ 3.657456] evm: Initialising EVM extended attributes: [ 3.659495] evm: security.selinux [ 3.660730] evm: security.ima [ 3.662263] evm: security.capability [ 3.663784] evm: HMAC attrs: 0x1 [ 3.666636] rtc_cmos 00:05: setting system clock to 2026-07-14 16:11:32 UTC (1784045492) [ 3.673660] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 3.677297] debug: unmapping init [mem 0xffffffff98b82000-0xffffffff98e58fff] [ 3.686097] Write protecting the kernel read-only data: 28672k [ 3.689979] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.693053] debug: unmapping init [mem 0xffffffff97b14000-0xffffffff97bfffff] [ 3.733627] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.742941] systemd[1]: Detected virtualization kvm. [ 3.745375] systemd[1]: Detected architecture x86-64. [ 3.747671] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.776218] systemd[1]: No hostname configured. [ 3.778464] systemd[1]: Set hostname to . [ 3.780875] random: systemd: uninitialized urandom read (16 bytes read) [ 3.783528] systemd[1]: Initializing machine ID from random generator. [ 3.842103] random: ln: uninitialized urandom read (6 bytes read) [ 3.937968] random: systemd: uninitialized urandom read (16 bytes read) [ 3.941151] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.946803] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.952502] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.645052] device-mapper: uevent: version 1.0.3 [ 4.647453] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.963231] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.375473] virtio_net virtio0 ens2: renamed from eth0 [ 5.494559] scsi host0: ata_piix [ 5.510557] scsi host1: ata_piix [ 5.512381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.515136] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.856329] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.169971] random: crng init done [ 10.171733] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.613232] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.761141] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.021381] SELinux: Disabled at runtime. [ 12.089673] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.100369] systemd[1]: Detected virtualization kvm. [ 12.102607] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.556735] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.562101] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.566723] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.571448] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.575168] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.582638] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.593678] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.738595] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [[ 13.099302] squashfs: version 4.0 (2009/01/31) Phillip Lougher  OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.450209] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.478371] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.560274] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.628749] EDAC sbridge: Ver: 1.1.2 [ 15.490729] Key type dns_resolver registered [ 15.800924] NFS: Registering the id_resolver key type [ 15.803207] Key type id_resolver registered [ 15.804520] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-server login: [ 55.187387] libcfs: loading out-of-tree module taints kernel. [ 55.224692] Key type ._llcrypt registered [ 55.228805] Key type .llcrypt registered [ 55.376975] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 76.321115] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 78.273284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.301833] alg: No test for adler32 (adler32-zlib) [ 78.807022] hrtimer: interrupt took 4118005 ns [ 80.107894] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 80.985436] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 82.825724] Key type lgssc registered [ 84.743048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.633108] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.956682] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 170.454925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.512072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.947296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.983787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.120421] Lustre: lustre-MDT0000: new disk, initializing [ 172.252714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.273874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.480122] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.698067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.877302] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.929478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.932688] Lustre: Skipped 1 previous similar message [ 196.019184] Lustre: lustre-MDT0001: new disk, initializing [ 196.063000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.138438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.159293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.914849] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.457662] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.941043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.267892] Lustre: lustre-OST0000: new disk, initializing [ 218.276824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.284185] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.366980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.562457] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.361793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.368461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 226.455138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.971468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.078838] Lustre: lustre-OST0001: new disk, initializing [ 242.081973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.086795] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.136531] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.427996] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.984819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.997942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.136653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 264.496453] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.246723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.056969] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.881407] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing yml_node [ 290.336189] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 293.116203] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 296.178624] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 297.844587] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 14 12:16:25 EDT 2026 [ 315.291392] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 316.712453] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.282318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.283765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.295205] Lustre: Skipped 1 previous similar message [ 338.405410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.423226] Lustre: Skipped 6 previous similar messages [ 338.861178] Lustre: server umount lustre-MDT0000 complete [ 342.625468] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045831 with bad export cookie 16342519400889816663 [ 342.629089] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.636783] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.544201] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.572041] Lustre: Skipped 2 previous similar messages [ 343.585764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.593425] Lustre: Skipped 1 previous similar message [ 347.553655] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.560470] Lustre: Skipped 1 previous similar message [ 349.057866] Lustre: server umount lustre-MDT0001 complete [ 353.239219] Lustre: server umount lustre-OST0000 complete [ 357.431941] Lustre: server umount lustre-OST0001 complete [ 363.976396] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 373.471620] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 410.737762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.057809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.099763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.205802] Lustre: lustre-MDT0000: new disk, initializing [ 411.352216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.380904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.429052] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 428.041960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.234110] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 428.310242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 428.314693] Lustre: Skipped 1 previous similar message [ 428.411943] Lustre: lustre-MDT0001: new disk, initializing [ 428.509080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.566502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.580155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.701049] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 442.254974] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 447.720294] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.244549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.659933] Lustre: lustre-OST0000: new disk, initializing [ 454.663512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 454.669090] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 454.764555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.281893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.298233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.400700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 460.641103] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 470.254838] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 475.068917] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 475.267904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.469524] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.726628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 487.397501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.398407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.407866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.410832] LustreError: Skipped 1 previous similar message [ 487.458242] Lustre: Skipped 2 previous similar messages [ 489.388242] Lustre: server umount lustre-OST0000 complete [ 495.075619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.087430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.634650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.641810] Lustre: Skipped 3 previous similar messages [ 499.402166] Lustre: server umount lustre-MDT0000 complete [ 502.754241] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.782975] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 503.169341] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045991 with bad export cookie 16342519400889818182 [ 503.172155] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.183834] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 503.522358] Lustre: server umount lustre-MDT0001 complete [ 508.558386] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:19:55 (1784045995) [ 510.873937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 513.029883] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:19:59 (1784045999) [ 520.372316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.927910] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.092562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.319402] LustreError: 19457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.340996] LustreError: 19457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 526.508019] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 530.410481] LustreError: 19456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.521969] LustreError: 19457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.059798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.114616] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 548.049463] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.634471] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 560.326760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 566.042957] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 573.103416] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 581.094804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.102269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 581.119600] Lustre: Skipped 1 previous similar message [ 581.122394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.148361] Lustre: Skipped 1 previous similar message [ 585.807920] Lustre: server umount lustre-OST0000 complete [ 591.331935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.355110] Lustre: Skipped 2 previous similar messages [ 595.792819] Lustre: server umount lustre-MDT0000 complete [ 596.450563] LustreError: 19453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.476532] LustreError: 19453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 599.103318] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046087 with bad export cookie 16342519400889819064 [ 599.106600] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.113322] LustreError: 19436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 599.504521] Lustre: server umount lustre-MDT0001 complete [ 610.358215] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 612.859944] Key type lgssc unregistered [ 613.139713] LNet: 22986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.144790] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.170194] LNet: Removed LNI 192.168.203.151@tcp [ 614.219188] Key type .llcrypt unregistered [ 614.222375] Key type ._llcrypt unregistered [ 635.664234] Key type ._llcrypt registered [ 635.668659] Key type .llcrypt registered [ 635.775875] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 637.097819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.124503] alg: No test for adler32 (adler32-zlib) [ 638.375921] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 638.574529] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 640.256172] Key type lgssc registered [ 641.504547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.351759] loop: module loaded [ 650.448912] loop0: detected capacity change from 0 to 204800000 [ 650.645320] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.657855] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.928596] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.854150] loop0: detected capacity change from 0 to 204800000 [ 657.031189] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 657.047379] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 657.061675] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.831923] loop0: detected capacity change from 0 to 204800000 [ 666.677366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 666.705551] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.008941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 668.094794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 668.207670] Lustre: lustre-MDT0000: new disk, initializing [ 668.373996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.404386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 673.440393] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 682.758378] loop1: detected capacity change from 0 to 204800000 [ 688.818782] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 689.184240] Lustre: lustre-OST07c6: new disk, initializing [ 689.192794] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 689.200689] Lustre: Skipped 1 previous similar message [ 689.205382] Lustre: 26174:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 689.302580] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 695.872403] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 697.916735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 697.926440] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 697.968473] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 700.410343] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 700.417389] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 700.427187] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 700.436320] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 700.441269] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 700.449279] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 714.989781] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 715.006538] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 715.010428] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 715.023047] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 715.032936] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 715.040482] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.347927] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 720.359240] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 720.365853] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.373778] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.381463] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.389494] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.395845] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 720.408551] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.421143] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 720.430497] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.438344] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.448794] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.379586] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 725.396371] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.407766] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 725.419138] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.433945] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.446204] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.455714] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 725.481254] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.499187] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.505714] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.517500] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.523195] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.244836] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 733.262127] Lustre: 25162:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 733.277801] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 733.288374] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 733.296419] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 733.312937] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 733.330893] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 733.339053] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 733.349838] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 733.354390] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 733.359808] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 733.364774] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 744.772063] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 744.777990] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 744.782073] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 744.789188] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 744.793249] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.798278] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 744.802400] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 744.805719] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 744.810773] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.815473] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 744.823199] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.831117] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.985791] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 789.994068] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.011863] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 794.683725] Lustre: server umount lustre-OST07c6 complete [ 803.491731] Lustre: server umount lustre-MDT0000 complete [ 822.510309] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 825.069386] Key type lgssc unregistered [ 825.273576] LNet: 28730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.287264] LNetError: 28730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.302241] LNet: Removed LNI 192.168.203.151@tcp [ 826.140889] Key type .llcrypt unregistered [ 826.143096] Key type ._llcrypt unregistered [ 849.468929] Key type ._llcrypt registered [ 849.470928] Key type .llcrypt registered [ 849.591705] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 850.530788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.585651] alg: No test for adler32 (adler32-zlib) [ 851.808494] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 852.123365] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 853.856683] Key type lgssc registered [ 854.636558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.002597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.035219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.491754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.458946] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 873.954287] LustreError: 30002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.966166] LustreError: 30002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 876.116289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.530565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.336038] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 888.117064] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 893.496914] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 899.238710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.652338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.503791] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 908.857524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 912.560626] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.201559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.219444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.248587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.254932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 929.257721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.265482] LustreError: Skipped 1 previous similar message [ 929.287758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.135327] Lustre: server umount lustre-OST0000 complete [ 937.953351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 937.965086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.982541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 937.991928] Lustre: Skipped 1 previous similar message [ 943.415697] Lustre: server umount lustre-MDT0000 complete [ 944.615113] LustreError: 30729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.635059] LustreError: 30729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 947.192149] LustreError: 29984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046436 with bad export cookie 11780961676002538133 [ 947.206307] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.207606] LustreError: 29984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 947.622512] Lustre: server umount lustre-MDT0001 complete [ 959.599220] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 962.618376] Key type lgssc unregistered [ 963.003713] LNet: 33739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.012569] LNetError: 33739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.052779] LNet: Removed LNI 192.168.203.151@tcp [ 964.059478] Key type .llcrypt unregistered [ 964.061485] Key type ._llcrypt unregistered [ 983.850793] Key type ._llcrypt registered [ 983.853769] Key type .llcrypt registered [ 983.985891] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 985.290611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.328888] alg: No test for adler32 (adler32-zlib) [ 986.425309] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 986.651295] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 988.321251] Key type lgssc registered [ 989.469635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 996.431375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.447946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 997.888713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.603676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1008.099688] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.129137] LustreError: 35017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1010.655269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.983707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1015.203093] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1021.970328] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1026.634876] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1032.570487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.972460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.419416] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1043.191720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1046.349317] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1056.739286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1056.752419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.772736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.790779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.794543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.809368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.421875] Lustre: server umount lustre-OST0000 complete [ 1067.489764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1067.502898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.519774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1067.526720] Lustre: Skipped 1 previous similar message [ 1072.022232] Lustre: server umount lustre-MDT0000 complete [ 1074.147569] LustreError: 35011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.179557] LustreError: 35011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1075.697404] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046564 with bad export cookie 15520957250016833805 [ 1075.702402] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.706523] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1076.058354] Lustre: server umount lustre-MDT0001 complete [ 1086.618677] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1089.127371] Key type lgssc unregistered [ 1089.492991] LNet: 38549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.498718] LNetError: 38549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.516628] LNet: Removed LNI 192.168.203.151@tcp [ 1090.307659] Key type .llcrypt unregistered [ 1090.309729] Key type ._llcrypt unregistered [ 1110.208853] Key type ._llcrypt registered [ 1110.212036] Key type .llcrypt registered [ 1110.381893] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1111.286442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.324948] alg: No test for adler32 (adler32-zlib) [ 1112.372847] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1112.587530] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1114.232178] Key type lgssc registered [ 1115.149727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1121.896479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1121.949204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.385727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.031862] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1133.539853] LustreError: 39946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.555716] LustreError: 39946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1133.749317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1134.007699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.683496] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1143.645145] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1148.501592] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1154.474467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1154.772886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.711381] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1166.158892] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1169.150441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1178.099922] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 1179.620078] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.623725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.637016] Lustre: Skipped 1 previous similar message [ 1179.647428] Lustre: Skipped 2 previous similar messages [ 1183.042497] Lustre: server umount lustre-MDT0000 complete [ 1183.201449] LustreError: 41038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.743601] LustreError: 41039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.767597] LustreError: 41039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1186.236280] LustreError: 39926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046675 with bad export cookie 14345326256666724128 [ 1186.240153] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.241866] LustreError: 39926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1188.312148] LustreError: 39942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.332160] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [ 1189.857733] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.873038] Lustre: Skipped 1 previous similar message [ 1191.914355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1191.922123] Lustre: Skipped 1 previous similar message [ 1192.611489] Lustre: server umount lustre-MDT0001 complete [ 1213.784303] Lustre: server umount lustre-OST0000 complete [ 1230.462666] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1232.576321] Key type lgssc unregistered [ 1232.887375] LNet: 43810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.893158] LNetError: 43810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.965761] LNet: Removed LNI 192.168.203.151@tcp [ 1234.596869] Key type .llcrypt unregistered [ 1234.599439] Key type ._llcrypt unregistered [ 1256.325828] Key type ._llcrypt registered [ 1256.327663] Key type .llcrypt registered [ 1256.406330] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1257.247792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.354308] alg: No test for adler32 (adler32-zlib) [ 1258.347591] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1258.609084] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1260.273090] Key type lgssc registered [ 1261.384282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1268.823533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1268.846884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.315617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.906822] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1280.482452] LustreError: 45225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.514216] LustreError: 45225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1281.943750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.442360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.382503] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1296.658371] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1302.920529] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1309.763953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.143658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1315.317044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1315.893929] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1324.030714] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1335.778359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1335.783862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.809972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.822462] Lustre: lustre-OST0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 1337.827149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1337.829299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1337.845830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.711738] Lustre: server umount lustre-OST0000 complete [ 1358.114759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.120782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.127673] Lustre: Skipped 1 previous similar message [ 1358.138650] Lustre: Skipped 2 previous similar messages [ 1363.425373] LustreError: 45955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.431841] Lustre: server umount lustre-MDT0000 complete [ 1363.434705] LustreError: 45955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1367.027289] LustreError: 48180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046855 with bad export cookie 16560689733259002542 [ 1367.031109] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.037078] LustreError: 48180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1381.344129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1381.715474] Lustre: server umount lustre-MDT0001 complete [ 1393.189561] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1396.157072] Key type lgssc unregistered [ 1396.475534] LNet: 48936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.481655] LNetError: 48936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.493062] LNet: Removed LNI 192.168.203.151@tcp [ 1397.255323] Key type .llcrypt unregistered [ 1397.256978] Key type ._llcrypt unregistered [ 1421.983399] Key type ._llcrypt registered [ 1421.986294] Key type .llcrypt registered [ 1422.091865] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1423.201544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.349534] alg: No test for adler32 (adler32-zlib) [ 1424.413880] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1424.669379] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1426.409644] Key type lgssc registered [ 1427.670291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1435.476069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1435.504809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.904679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.713284] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1446.377418] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.390723] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1450.747542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.271869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1456.065875] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1464.120455] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1469.784208] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1476.922588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.097773] LustreError: 52172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.388945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.522814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1484.018121] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1491.205542] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1497.681790] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1501.446591] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1513.791620] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-1412h36m40s [ 1515.832870] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1518.050559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1518.064188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.083239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.144599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.148175] Lustre: Skipped 3 previous similar messages [ 1520.675924] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 1520.694775] Lustre: Skipped 2 previous similar messages [ 1525.236175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1525.258799] Lustre: Skipped 2 previous similar messages [ 1531.378293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.387667] Lustre: Skipped 1 previous similar message [ 1532.388914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1532.500669] Lustre: server umount lustre-MDT0000 complete [ 1536.174660] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1537.507058] LustreError: 51896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.527067] LustreError: 51896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1543.654897] LustreError: 51079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.666925] LustreError: 51079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1544.045810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.215192] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1544.587566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.720754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1545.585402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1549.335279] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1552.725933] Lustre: DEBUG MARKER: Wait for df (36673) ... [ 1554.657740] Lustre: DEBUG MARKER: done [ 1560.053586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1560.068334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.089713] Lustre: Skipped 2 previous similar messages [ 1560.096835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.099602] Lustre: Skipped 1 previous similar message [ 1565.526442] Lustre: server umount lustre-OST0000 complete [ 1571.300308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1571.307086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.322811] Lustre: Skipped 2 previous similar messages [ 1575.981516] Lustre: server umount lustre-MDT0000 complete [ 1578.468408] LustreError: 50352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.489405] LustreError: 50352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1579.764383] LustreError: 50331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047068 with bad export cookie 10716320894907609730 [ 1579.767540] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.774118] LustreError: 50331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1580.260241] Lustre: server umount lustre-MDT0001 complete [ 1592.039579] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1595.086485] Key type lgssc unregistered [ 1595.525288] LNet: 55274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.544836] LNetError: 55274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.586091] LNet: Removed LNI 192.168.203.151@tcp [ 1597.615268] Key type .llcrypt unregistered [ 1597.618866] Key type ._llcrypt unregistered [ 1641.436455] Key type ._llcrypt registered [ 1641.440352] Key type .llcrypt registered [ 1641.596135] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1643.033631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1643.174718] alg: No test for adler32 (adler32-zlib) [ 1644.581917] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1644.986318] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1646.768172] Key type lgssc registered [ 1647.774463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1657.157993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.216391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.559482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1658.574314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1658.703631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.120388] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1672.681868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.845439] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1672.876683] Lustre: 57485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1673.110679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1677.922578] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1684.958491] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1691.062366] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1698.163771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1698.435963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1698.665919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.939098] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1705.656434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1711.783700] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1719.000038] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-1412h40m06s [ 1720.383550] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1727.219852] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1729.761361] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1729.806564] LustreError: 57492:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1750.048745] Lustre: lustre-MDT0000: Client 5b1f2f7c-7ec1-4ea0-8a7e-0fe960cfdf5d (at 192.168.203.51@tcp) reconnecting [ 1769.819413] LustreError: 57492:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1772.064065] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1774.726281] Lustre: DEBUG MARKER: done [ 1782.241534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.260802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.278175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.434805] Lustre: server umount lustre-OST0000 complete [ 1787.877767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.880845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.889022] Lustre: Skipped 1 previous similar message [ 1787.912021] Lustre: Skipped 1 previous similar message [ 1788.962520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.969295] Lustre: Skipped 1 previous similar message [ 1794.021425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.036347] Lustre: Skipped 1 previous similar message [ 1796.146441] Lustre: server umount lustre-MDT0000 complete [ 1799.620556] LustreError: 60246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047288 with bad export cookie 2285067315266839975 [ 1799.637625] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.045919] Lustre: server umount lustre-MDT0001 complete [ 1810.081764] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 1812.909504] Key type lgssc unregistered [ 1813.206721] LNet: 61817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.223549] LNetError: 61817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.241934] LNet: Removed LNI 192.168.203.151@tcp [ 1814.046946] Key type .llcrypt unregistered [ 1814.053177] Key type ._llcrypt unregistered [ 1856.741376] Key type ._llcrypt registered [ 1856.744312] Key type .llcrypt registered [ 1856.844689] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 1858.084945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1858.101191] alg: No test for adler32 (adler32-zlib) [ 1859.224944] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 1859.554954] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 1861.227335] Key type lgssc registered [ 1862.600786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1866.267216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1866.283811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.644985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1867.699473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1868.035212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.430585] Lustre: Failing over lustre-MDT0000 [ 1869.876785] Lustre: server umount lustre-MDT0000 complete [ 1877.340958] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:42:44 (1784047364) [ 1885.513555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.868167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.616210] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1900.046695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.124787] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1900.167074] Lustre: 64613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1900.485648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.552924] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1913.689825] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1920.170787] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.478066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.740035] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1927.960288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.494078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1935.132492] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 1944.011411] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.081522] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1959.405761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.409786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.431254] Lustre: Skipped 2 previous similar messages [ 1959.436164] Lustre: Skipped 1 previous similar message [ 1964.518051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.524957] Lustre: Skipped 3 previous similar messages [ 1965.097396] Lustre: server umount lustre-MDT0000 complete [ 1969.094846] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047457 with bad export cookie 15195503281355251922 [ 1969.103764] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.107304] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1969.304407] Lustre: server umount lustre-MDT0001 complete [ 1972.716330] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1978.552377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1986.017808] Lustre: 63536:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047458/real 1784047458] req@00000000c604c37b x1870709232018432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1986.065716] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.819089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.136611] Lustre: 63537:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047463/real 1784047463] req@000000006ae037dc x1870709232018560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1994.295486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.445181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1995.232984] Lustre: 63538:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047467/real 1784047467] req@00000000703e872d x1870709232018816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1999.331420] Lustre: 63536:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047472/real 1784047472] req@0000000026b81e06 x1870709232018944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047488 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.006873] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2000.359510] LustreError: 68386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.386410] LustreError: 68386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2005.481779] LustreError: 66464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.526927] LustreError: 66464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2010.263838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.596089] LustreError: 68385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.614951] LustreError: 68385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2010.908050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.028704] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2016.494714] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2024.778698] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2032.182730] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2040.816181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.822061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2040.829511] Lustre: Skipped 2 previous similar messages [ 2040.840391] Lustre: Skipped 2 previous similar messages [ 2045.924290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.932769] Lustre: Skipped 2 previous similar messages [ 2046.416510] Lustre: server umount lustre-MDT0000 complete [ 2050.587641] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047539 with bad export cookie 15195503281355254197 [ 2050.590196] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.600944] LustreError: 66469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2050.897761] Lustre: server umount lustre-MDT0001 complete [ 2057.813962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2066.400345] Lustre: 63537:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047539/real 1784047539] req@000000008b9d670b x1870709232046592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.437833] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.796893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.033697] LustreError: 66464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.089820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.142112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2077.088319] Lustre: 63535:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784047549/real 1784047549] req@00000000350587d7 x1870709232047104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784047565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2077.137798] Lustre: 63535:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2080.924706] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2082.289896] LustreError: 71067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.314437] LustreError: 71067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2090.035198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.284456] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2095.600455] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2095.608353] Lustre: Skipped 1 previous similar message [ 2096.686603] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2103.150681] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.393096] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2115.064362] Lustre: server umount lustre-MDT0000 complete [ 2116.067278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2116.088641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.108950] LustreError: 71801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.123248] LustreError: 71801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2119.063975] LustreError: 64604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047607 with bad export cookie 15195503281355255576 [ 2119.080258] LustreError: 64604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2119.084241] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.525767] Lustre: server umount lustre-MDT0001 complete [ 2125.403950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.694658] Lustre: DEBUG MARKER: files compared the same [ 2140.986943] Lustre: server umount lustre-OST0000 complete [ 2158.008697] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2160.803863] Key type lgssc unregistered [ 2161.096943] LNet: 74341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.105535] LNetError: 74341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.153991] LNet: Removed LNI 192.168.203.151@tcp [ 2163.119178] Key type .llcrypt unregistered [ 2163.122444] Key type ._llcrypt unregistered [ 2189.673883] Key type ._llcrypt registered [ 2189.682617] Key type .llcrypt registered [ 2189.813701] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2191.359734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.392588] alg: No test for adler32 (adler32-zlib) [ 2192.570189] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2192.908523] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2194.641206] Key type lgssc registered [ 2195.924836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2204.401526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2204.453936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.139550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.627258] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug +malloc all [ 2215.396349] LustreError: 75760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.413913] LustreError: 75760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2219.195773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.679666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2225.486582] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug +malloc all [ 2233.301802] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.130226] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2246.400578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.627272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2250.755394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2253.439820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug +malloc all [ 2262.872365] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.427223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.427563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.445667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.472760] Lustre: Skipped 1 previous similar message [ 2282.624585] Lustre: server umount lustre-OST0000 complete [ 2290.657391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.669283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.669407] Lustre: Skipped 2 previous similar messages [ 2290.687454] Lustre: Skipped 1 previous similar message [ 2294.884876] Lustre: server umount lustre-MDT0000 complete [ 2295.783203] LustreError: 75759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.799835] LustreError: 75759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2298.877947] LustreError: 75740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047787 with bad export cookie 6811910273598055223 [ 2298.885239] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.903037] LustreError: 75740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2299.365936] Lustre: server umount lustre-MDT0001 complete [ 2313.203117] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2316.444294] Key type lgssc unregistered [ 2316.772208] LNet: 79315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.782167] LNetError: 79315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.807244] LNet: Removed LNI 192.168.203.151@tcp [ 2317.832850] Key type .llcrypt unregistered [ 2317.833995] Key type ._llcrypt unregistered [ 2344.125435] Key type ._llcrypt registered [ 2344.131782] Key type .llcrypt registered [ 2344.288556] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2346.042648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.083079] alg: No test for adler32 (adler32-zlib) [ 2347.321831] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2347.666113] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2349.392192] Key type lgssc registered [ 2350.984210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2360.601346] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2360.644441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2376.226515] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2381.344401] LustreError: 80708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [ 2407.975859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.050380] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2424.152763] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2433.218350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.389974] Lustre: *** cfs_fail_loc=706, val=0*** [ 2433.769033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.895816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2438.824030] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2439.141388] LustreError: 80733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.169831] LustreError: 80733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2442.211737] LustreError: 80734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.243580] LustreError: 80734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2447.338920] LustreError: 80733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.367539] LustreError: 80733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2448.333885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.688167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2453.783768] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2462.242527] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2468.237112] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.471567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.474459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2474.479852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2474.486515] Lustre: Skipped 1 previous similar message [ 2476.963743] Lustre: server umount lustre-OST0000 complete [ 2484.712656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.715127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.733268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.733276] Lustre: Skipped 1 previous similar message [ 2484.774522] Lustre: Skipped 1 previous similar message [ 2487.669490] Lustre: server umount lustre-MDT0000 complete [ 2489.825380] LustreError: 81798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.251480] LustreError: 80716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047981 with bad export cookie 3826658966712661238 [ 2492.260550] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.264539] LustreError: 80716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2492.773456] Lustre: server umount lustre-MDT0001 complete [ 2507.537635] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2510.580264] Key type lgssc unregistered [ 2510.996731] LNet: 84324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.001911] LNetError: 84324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.022680] LNet: Removed LNI 192.168.203.151@tcp [ 2511.942253] Key type .llcrypt unregistered [ 2511.946365] Key type ._llcrypt unregistered [ 2544.321462] Key type ._llcrypt registered [ 2544.324395] Key type .llcrypt registered [ 2544.448352] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2545.898329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.946264] alg: No test for adler32 (adler32-zlib) [ 2547.261541] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2547.556167] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2549.280353] Key type lgssc registered [ 2550.675912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2559.537508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2559.552868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.214270] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2574.905234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.621446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.970363] LustreError: 86431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.989657] LustreError: 86431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2581.035517] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2586.081785] LustreError: 86434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.094360] LustreError: 86434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2590.577328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2590.958276] LustreError: 86431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.996492] LustreError: 86431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2591.020344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2596.322834] LustreError: 86435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.363402] LustreError: 86435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2598.940676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2600.419814] LustreError: 86429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.446285] LustreError: 86429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2608.519921] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.541986] Lustre: MGS: Client ea8c7ed4-8721-411f-a127-f99292acb967 (at 0@lo) reconnecting [ 2608.585933] LustreError: 86445:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2608.790634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.871786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2613.810944] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2629.608952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.610123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.632224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.736460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.751099] Lustre: Skipped 1 previous similar message [ 2635.246449] Lustre: server umount lustre-OST0000 complete [ 2639.840872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.858790] Lustre: Skipped 2 previous similar messages [ 2639.862441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.862893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.886876] Lustre: Skipped 2 previous similar messages [ 2644.962304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.973600] Lustre: Skipped 1 previous similar message [ 2645.822848] Lustre: server umount lustre-MDT0000 complete [ 2650.642321] Lustre: server umount lustre-MDT0001 complete [ 2655.381446] Lustre: server umount MGS complete [ 2669.087976] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:55:55 (1784048155) [ 2704.586602] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 2717.476973] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2760.254971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.329603] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2775.463686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.639203] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2775.684414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2775.811780] Lustre: lustre-MDT0001: new disk, initializing [ 2775.926055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.944638] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2775.949296] Lustre: 93623:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2775.954013] LustreError: 93623:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2780.824402] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2794.197717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.604315] Lustre: lustre-OST0000: new disk, initializing [ 2794.610254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2794.620135] Lustre: 94390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2794.723869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.946931] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2805.859577] Lustre: 94847:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000640b1a16 x1870709953437696/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784048310 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2812.088197] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2819.389364] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.401899] Lustre: MGS: Client 81ae6981-e1a6-41bd-bc4f-74697ada4772 (at 0@lo) reconnecting [ 2819.507452] Lustre: 92892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2819.523498] LustreError: 93609:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2819.562327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2819.675229] Lustre: lustre-MDT0000: new disk, initializing [ 2819.884650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.927306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2820.043697] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2820.962776] Lustre: 95460:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000003c41435e x1870709953444352/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784048325 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2824.773738] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2826.209430] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2830.125845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2830.137895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2830.216859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2855.912771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.930944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.941834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.957687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.984259] Lustre: server umount lustre-OST0000 complete [ 2868.195283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.198293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.213408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.239629] Lustre: Skipped 3 previous similar messages [ 2872.443982] Lustre: server umount lustre-MDT0000 complete [ 2873.314702] LustreError: 93599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.333235] LustreError: 93599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2891.232271] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2891.585255] Lustre: server umount lustre-MDT0001 complete [ 2895.926088] Lustre: server umount MGS complete [ 2910.543648] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:59:57 (1784048397) [ 2933.061445] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 2936.795901] Key type lgssc unregistered [ 2937.278765] LNet: 97964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.289239] LNetError: 97964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2937.309191] LNet: Removed LNI 192.168.203.151@tcp [ 2938.231908] Key type .llcrypt unregistered [ 2938.235890] Key type ._llcrypt unregistered [ 2959.573983] Key type ._llcrypt registered [ 2959.579980] Key type .llcrypt registered [ 2959.689923] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 2960.916986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2961.029049] alg: No test for adler32 (adler32-zlib) [ 2962.218553] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 2962.569688] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 2964.336229] Key type lgssc registered [ 2965.804642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2975.165537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2975.218396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.667067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.763652] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 2986.996568] LustreError: 99199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.028591] LustreError: 99199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2991.989229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.103413] LustreError: 99198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.535967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2997.783567] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3007.607473] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3015.723361] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3023.822437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.246745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.023312] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3041.696876] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3046.771344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3057.137073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.173872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3057.175156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3062.223654] Lustre: server umount lustre-OST0000 complete [ 3067.362278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.376611] Lustre: Skipped 2 previous similar messages [ 3067.379767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3067.394622] Lustre: Skipped 2 previous similar messages [ 3072.485353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.491948] Lustre: Skipped 1 previous similar message [ 3077.602214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3077.613029] Lustre: Skipped 1 previous similar message [ 3081.184150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3081.397478] Lustre: server umount lustre-MDT0000 complete [ 3082.729374] LustreError: 100176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3082.751662] LustreError: 100176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3085.830885] LustreError: 99178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048574 with bad export cookie 4353939479266504279 [ 3085.838925] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.842197] LustreError: 99178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3086.185236] Lustre: server umount lustre-MDT0001 complete [ 3099.795411] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3102.919597] Key type lgssc unregistered [ 3103.273108] LNet: 102756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3103.302419] LNetError: 102756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3104.360220] LNet: Removed LNI 192.168.203.151@tcp [ 3105.400903] Key type .llcrypt unregistered [ 3105.410161] Key type ._llcrypt unregistered [ 3122.587548] Key type ._llcrypt registered [ 3122.590306] Key type .llcrypt registered [ 3122.659070] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3123.490947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.598062] alg: No test for adler32 (adler32-zlib) [ 3124.706687] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 3125.050219] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3126.738576] Key type lgssc registered [ 3128.254792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.630238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.641737] LustreError: 103837:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3135.662890] LustreError: 103841:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3135.676500] LustreError: 103837:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3135.679823] LustreError: 103837:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3135.697076] LustreError: 103837:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3135.716694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.188152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.541393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.942479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3150.890626] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3156.052464] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing lsmod [ 3159.220426] LustreError: 105047:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3159.256124] LustreError: 105051:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3159.261095] LustreError: 105047:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 3159.269861] LustreError: 105047:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3159.275698] LustreError: 105047:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3159.280259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3159.687798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.159434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.161542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.168255] Lustre: Skipped 1 previous similar message [ 3165.185314] Lustre: Skipped 2 previous similar messages [ 3168.656731] Lustre: server umount lustre-MDT0000 complete [ 3170.277440] LustreError: 104399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.291636] LustreError: 104399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3172.162781] LustreError: 103855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048660 with bad export cookie 8198025299556837561 [ 3172.172170] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.177263] LustreError: 103855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3172.365219] LustreError: 104311:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e3d339be x1870710558444928/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3172.566261] Lustre: server umount lustre-MDT0001 complete [ 3186.842922] Lustre: server umount lustre-OST0000 complete [ 3195.123237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.498269] LustreError: 106114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.564658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.213869] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3200.999229] LustreError: 106115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.117395] LustreError: 106114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.449303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.104900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.713985] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3220.995099] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3226.998181] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3234.313235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3234.723042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.908058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3242.222282] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3250.476864] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3263.985134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3263.992129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.018547] Lustre: Skipped 1 previous similar message [ 3264.038364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.577951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.583744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.584788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.594474] Lustre: Skipped 1 previous similar message [ 3270.325291] Lustre: server umount lustre-OST0000 complete [ 3278.819889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.826340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.826878] Lustre: Skipped 1 previous similar message [ 3281.242558] Lustre: server umount lustre-MDT0000 complete [ 3283.939686] LustreError: 107118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.976934] LustreError: 107118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3284.880365] LustreError: 106096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048773 with bad export cookie 8198025299556838261 [ 3284.904151] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3284.906893] LustreError: 106096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3285.135729] Lustre: server umount lustre-MDT0001 complete [ 3302.599532] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3306.226236] Key type lgssc unregistered [ 3306.561105] LNet: 109689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.573406] LNetError: 109689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.597675] LNet: Removed LNI 192.168.203.151@tcp [ 3307.571100] Key type .llcrypt unregistered [ 3307.575158] Key type ._llcrypt unregistered [ 3336.043566] Key type ._llcrypt registered [ 3336.058675] Key type .llcrypt registered [ 3336.205641] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3337.432470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3337.598657] alg: No test for adler32 (adler32-zlib) [ 3338.721822] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 3339.117345] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3340.888369] Key type lgssc registered [ 3342.260228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3351.708403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3351.733320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.105561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.320217] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3362.284767] LustreError: 111106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.300340] LustreError: 111106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3367.305592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.408571] LustreError: 111105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.919703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.681414] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3381.457654] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3390.226784] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.330455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.902751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.616332] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3418.510110] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3422.435478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3430.206899] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3434.322438] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3439.075207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3439.082694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.115191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3443.169367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3443.175490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3443.185474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.217767] Lustre: Skipped 1 previous similar message [ 3445.317474] Lustre: server umount lustre-OST0000 complete [ 3449.828813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3449.844494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.881927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.410095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.411597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.428856] Lustre: Skipped 1 previous similar message [ 3455.767120] Lustre: server umount lustre-MDT0000 complete [ 3458.530680] LustreError: 111106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3458.563241] LustreError: 111106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3460.159812] LustreError: 111087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048948 with bad export cookie 241529139165191404 [ 3460.163177] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3460.171641] LustreError: 111087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3460.551627] Lustre: server umount lustre-MDT0001 complete [ 3473.075361] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3476.323310] Key type lgssc unregistered [ 3476.679298] LNet: 114765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.691939] LNetError: 114765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3476.735665] LNet: Removed LNI 192.168.203.151@tcp [ 3477.887137] Key type .llcrypt unregistered [ 3477.891955] Key type ._llcrypt unregistered [ 3496.541412] Key type ._llcrypt registered [ 3496.544612] Key type .llcrypt registered [ 3496.708604] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3498.108168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.267817] alg: No test for adler32 (adler32-zlib) [ 3499.348112] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 3499.734294] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3501.464184] Key type lgssc registered [ 3502.837794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3511.438585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3511.450236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.935550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.196886] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3522.529891] LustreError: 115988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.562831] LustreError: 115988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3527.650124] LustreError: 115989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.684150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.056924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3533.442112] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3542.469874] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3548.790293] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3555.241820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.610181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.698991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3562.335771] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3570.744534] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3583.462433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.462878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3583.494016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.594965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.597671] Lustre: Skipped 1 previous similar message [ 3593.698577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.715355] Lustre: Skipped 1 previous similar message [ 3594.208171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3594.586662] Lustre: server umount lustre-OST0000 complete [ 3599.843890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3599.850881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.863879] Lustre: Skipped 1 previous similar message [ 3599.867656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3603.937642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3603.941171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.525991] Lustre: server umount lustre-MDT0000 complete [ 3609.058659] LustreError: 115988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.076036] LustreError: 115988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3609.231728] LustreError: 115971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049098 with bad export cookie 16694117337784541621 [ 3609.239503] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3609.239522] LustreError: 115971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3609.509387] Lustre: server umount lustre-MDT0001 complete [ 3620.706884] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3623.676577] Key type lgssc unregistered [ 3623.928837] LNet: 119524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3623.940351] LNetError: 119524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3623.957843] LNet: Removed LNI 192.168.203.151@tcp [ 3624.725489] Key type .llcrypt unregistered [ 3624.727204] Key type ._llcrypt unregistered [ 3644.139395] Key type ._llcrypt registered [ 3644.142619] Key type .llcrypt registered [ 3644.232720] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3645.248800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3645.270514] alg: No test for adler32 (adler32-zlib) [ 3646.309720] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 3646.544245] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3648.208776] Key type lgssc registered [ 3649.318897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3655.945361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3655.973667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.499969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.458591] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3667.942237] LustreError: 120922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3667.959478] LustreError: 120922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3670.618463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.990371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3675.565569] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3682.999174] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3688.679629] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3695.552192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3695.955617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.620911] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3703.159031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3710.674600] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3721.716093] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3726.035732] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3726.046068] Lustre: Skipped 2 previous similar messages [ 3727.401159] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3727.423619] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3730.548354] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3730.553628] Lustre: Skipped 2 previous similar messages [ 3735.151496] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3735.153604] Lustre: Skipped 1 previous similar message [ 3739.586229] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3739.594819] Lustre: Skipped 2 previous similar messages [ 3740.444104] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.51@tcp 192.168.0.[2,10]@tcp [ 3745.183199] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.51/24@tcp in log lustre-MDT0000 [ 3745.191710] Lustre: Skipped 2 previous similar messages [ 3745.878792] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.51/24@tcp [ 3745.887299] Lustre: Skipped 1 previous similar message [ 3759.591540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3759.597861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.625626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.640082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.654834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.662558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.388541] Lustre: server umount lustre-OST0000 complete [ 3771.874233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.893700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.913115] Lustre: Skipped 1 previous similar message [ 3776.993410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3777.000345] Lustre: Skipped 1 previous similar message [ 3782.113906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3782.123461] Lustre: Skipped 1 previous similar message [ 3784.160223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3784.375873] Lustre: server umount lustre-MDT0000 complete [ 3787.233650] LustreError: 120915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.256628] LustreError: 120915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3788.305635] LustreError: 120903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049277 with bad export cookie 10207358238778904826 [ 3788.307945] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3788.322240] LustreError: 120903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3788.761818] Lustre: server umount lustre-MDT0001 complete [ 3799.367445] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3802.275552] Key type lgssc unregistered [ 3802.638829] LNet: 125580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3802.648203] LNetError: 125580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3802.671519] LNet: Removed LNI 192.168.203.151@tcp [ 3803.703554] Key type .llcrypt unregistered [ 3803.706565] Key type ._llcrypt unregistered [ 3833.958531] Key type ._llcrypt registered [ 3833.964868] Key type .llcrypt registered [ 3834.096994] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 3835.477067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3835.547126] alg: No test for adler32 (adler32-zlib) [ 3836.624843] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 3836.844525] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 3838.512235] Key type lgssc registered [ 3839.641739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3847.765482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3847.786772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.318664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.327218] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3849.373431] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.51/24@tcp [ 3853.898862] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3859.457952] LustreError: 127145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3859.483552] LustreError: 127145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3864.552390] LustreError: 127146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.807184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.369891] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3865.386816] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.51/24@tcp [ 3865.496593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.719154] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3881.190635] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3888.729635] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3896.367447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3896.730282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.166910] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 3906.278979] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3919.309291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3939.812958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3939.821380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.829601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.857832] Lustre: Skipped 1 previous similar message [ 3943.667584] Lustre: server umount lustre-OST0000 complete [ 3950.048997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3950.054455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3950.067351] Lustre: Skipped 1 previous similar message [ 3950.094965] Lustre: Skipped 2 previous similar messages [ 3954.800437] Lustre: server umount lustre-MDT0000 complete [ 3955.215132] LustreError: 127140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.253708] LustreError: 127140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3959.330321] LustreError: 128411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.351839] LustreError: 128411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3960.620343] LustreError: 127126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049449 with bad export cookie 1036183646240335537 [ 3960.631887] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.635675] LustreError: 127126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3961.614852] Lustre: server umount lustre-MDT0001 complete [ 3976.412244] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 3979.778753] Key type lgssc unregistered [ 3980.129460] LNet: 130760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3980.139734] LNetError: 130760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3980.161870] LNet: Removed LNI 192.168.203.151@tcp [ 3981.601160] Key type .llcrypt unregistered [ 3981.604212] Key type ._llcrypt unregistered [ 4008.621026] Key type ._llcrypt registered [ 4008.623431] Key type .llcrypt registered [ 4008.808948] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4026.138851] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4027.487145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4027.511794] alg: No test for adler32 (adler32-zlib) [ 4028.564447] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 4028.867363] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 4030.568322] Key type lgssc registered [ 4031.884510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4067.532323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4067.557246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.754623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4068.788341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4068.860491] Lustre: lustre-MDT0000: new disk, initializing [ 4068.915656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.945580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4073.727735] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4084.394504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.515131] Lustre: 134161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4084.577521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4084.583862] Lustre: Skipped 1 previous similar message [ 4084.683795] Lustre: lustre-MDT0001: new disk, initializing [ 4084.786682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4084.815951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4084.831694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4089.590827] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4099.526909] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4106.525138] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4113.846193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4114.140361] Lustre: lustre-OST0000: new disk, initializing [ 4114.143575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4114.148707] Lustre: 136128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4114.213317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.659729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4119.674438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4119.846344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4120.381243] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4131.196362] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4137.032965] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4137.387830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4142.273047] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4142.481393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4145.644491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4145.646880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.662692] LustreError: Skipped 1 previous similar message [ 4145.681980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4145.695117] Lustre: Skipped 1 previous similar message [ 4150.760343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4150.767386] Lustre: Skipped 1 previous similar message [ 4151.023934] Lustre: server umount lustre-OST0000 complete [ 4155.874300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.899164] Lustre: Skipped 1 previous similar message [ 4155.904803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.915799] Lustre: Skipped 1 previous similar message [ 4161.163834] Lustre: server umount lustre-MDT0000 complete [ 4164.685392] LustreError: 134152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049653 with bad export cookie 10093150953090722283 [ 4164.690826] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.695196] LustreError: 134152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4165.042773] Lustre: server umount lustre-MDT0001 complete [ 4172.902562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.249808] LustreError: 138347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.274878] LustreError: 138347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4173.355841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.270681] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4178.402071] LustreError: 138348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4183.523726] LustreError: 138347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.278073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.263208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4193.039534] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4202.543094] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4209.373967] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4216.216649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4216.520954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.740166] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4231.309838] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4236.198963] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4236.533972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4241.794174] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4241.967709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4252.505664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.815724] Lustre: lustre-OST0001: new disk, initializing [ 4252.827787] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4252.844597] Lustre: 141720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4252.953815] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4258.678760] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4260.471986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4260.484651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4260.666464] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4267.043541] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4267.203851] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4271.833560] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4272.074497] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.568038] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f6902f8e000.ost_server_uuid 50 [ 4282.018780] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f6902f8e000.ost_server_uuid in IDLE state after 0 sec [ 4296.688463] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4296.695102] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4296.724973] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4296.727833] LustreError: Skipped 1 previous similar message [ 4296.778211] Lustre: Skipped 2 previous similar messages [ 4299.993097] Lustre: server umount lustre-OST0001 complete [ 4301.799349] LustreError: 140172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.820106] LustreError: 140172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4305.891275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4305.898164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.914434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4305.917179] Lustre: Skipped 1 previous similar message [ 4306.916516] LustreError: 140172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.949376] LustreError: 140172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4310.744908] Lustre: server umount lustre-OST0000 complete [ 4315.106561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4315.109650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.117070] Lustre: Skipped 1 previous similar message [ 4315.123949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4315.131574] Lustre: Skipped 2 previous similar messages [ 4320.574228] Lustre: server umount lustre-MDT0000 complete [ 4322.282919] LustreError: 138347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.296663] LustreError: 138347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4323.717639] LustreError: 138329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784049812 with bad export cookie 10093150953090723165 [ 4323.720671] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.721459] LustreError: 138329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4324.015566] Lustre: server umount lustre-MDT0001 complete [ 4342.495915] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 4345.023844] Key type lgssc unregistered [ 4345.265681] LNet: 144722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4345.275721] LNetError: 144722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4345.283705] LNet: Removed LNI 192.168.203.151@tcp [ 4346.060334] Key type .llcrypt unregistered [ 4346.063438] Key type ._llcrypt unregistered [ 4389.642891] Key type ._llcrypt registered [ 4389.644864] Key type .llcrypt registered [ 4389.763253] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4411.929555] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4413.139655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4413.219122] alg: No test for adler32 (adler32-zlib) [ 4414.381602] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 4414.575802] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 4416.200184] Key type lgssc registered [ 4417.338105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4451.392186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4451.410525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.622789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4452.643285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4452.711983] Lustre: lustre-MDT0000: new disk, initializing [ 4452.801969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.817497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4457.873073] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4473.164982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.382573] Lustre: 148886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4473.508967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4473.517959] Lustre: Skipped 1 previous similar message [ 4473.665748] Lustre: lustre-MDT0001: new disk, initializing [ 4473.748184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4473.810943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4473.826068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4480.626545] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4492.159524] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4498.291946] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4505.321434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4505.506204] Lustre: lustre-OST0000: new disk, initializing [ 4505.512540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4505.524123] Lustre: 150859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4505.611756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.574031] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4514.398411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4514.417150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4514.549685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4522.370517] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4527.485105] Lustre: Failing over lustre-OST0000 [ 4527.635505] Lustre: server umount lustre-OST0000 complete [ 4545.504858] Lustre: 147111:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050018/real 1784050018] req@000000004c5347d3 x1870711911123712/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784050034 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4545.523531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.533512] Lustre: 147111:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4545.557426] Lustre: Skipped 1 previous similar message [ 4547.230059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4547.716279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4547.756035] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4551.691221] Lustre: 147112:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050024/real 1784050024] req@0000000005f7af86 x1870711911124992/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784050040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4551.731895] Lustre: 147112:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4554.355464] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4555.758586] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4555.944930] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4555.945116] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4556.962495] Lustre: 147113:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050029/real 1784050029] req@000000006fa01330 x1870711911126016/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784050045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4557.012752] Lustre: 147113:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4559.916946] Lustre: Failing over lustre-MDT0000 [ 4560.516470] Lustre: server umount lustre-MDT0000 complete [ 4560.867506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4560.885912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.979434] LustreError: 148896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.002158] LustreError: 148896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4566.499075] LustreError: 148900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.526805] LustreError: 148900:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4571.112114] LustreError: 148895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4576.230805] LustreError: 148895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4576.249255] LustreError: 148895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4577.633490] Lustre: 147111:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784050050/real 1784050050] req@0000000005f7af86 x1870711911136384/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784050066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4577.668267] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.342996] LustreError: 148895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.373744] LustreError: 148895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4581.591408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.000680] LustreError: 147109:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e65ba113 x1870711911141504/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4588.275394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.341266] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4591.583961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4592.214538] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4593.658614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4593.673098] Lustre: Skipped 1 previous similar message [ 4593.717181] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4593.792466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4602.340232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4602.348258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.356358] Lustre: Skipped 2 previous similar messages [ 4602.359989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4603.877310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4608.513776] Lustre: server umount lustre-OST0000 complete [ 4614.113362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4614.120058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.131588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.139080] Lustre: Skipped 2 previous similar messages [ 4614.159423] Lustre: Skipped 2 previous similar messages [ 4619.237153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4619.248466] Lustre: Skipped 1 previous similar message [ 4620.208985] Lustre: server umount lustre-MDT0000 complete [ 4623.844231] LustreError: 148878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050112 with bad export cookie 3860098976561856919 [ 4623.850398] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4623.870942] LustreError: 148878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4624.178371] Lustre: server umount lustre-MDT0001 complete [ 4635.431397] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 4638.301178] Key type lgssc unregistered [ 4638.600933] LNet: 154519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4638.609355] LNetError: 154519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4638.632221] LNet: Removed LNI 192.168.203.151@tcp [ 4639.648954] Key type .llcrypt unregistered [ 4639.654506] Key type ._llcrypt unregistered [ 4678.728120] Key type ._llcrypt registered [ 4678.740562] Key type .llcrypt registered [ 4678.852972] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4680.710503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4680.839691] alg: No test for adler32 (adler32-zlib) [ 4681.954722] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 4682.159274] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 4683.840577] Key type lgssc registered [ 4685.126544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4693.177578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4693.212177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.779887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.284744] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4704.230749] LustreError: 156039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.246425] LustreError: 156039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4709.012669] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4709.351646] LustreError: 156038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.011468] LustreError: 156067:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4713.016705] LustreError: 156067:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4713.021947] LustreError: 156067:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4713.322725] Lustre: server umount lustre-MDT0000 complete [ 4729.316385] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 4740.792381] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 4780.040031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.253386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4780.279253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4780.347919] Lustre: lustre-MDT0000: new disk, initializing [ 4780.416931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4780.434721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4784.747370] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4796.782979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.120733] Lustre: 159371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4797.186911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4797.195117] Lustre: Skipped 1 previous similar message [ 4797.271236] Lustre: lustre-MDT0001: new disk, initializing [ 4797.450332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4797.515581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4797.542757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4802.292399] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4812.819234] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4819.040477] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4825.895209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4826.113385] Lustre: lustre-OST0000: new disk, initializing [ 4826.117384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4826.123927] Lustre: 161346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4826.224161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.087639] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 4834.390215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4834.400330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4834.531613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4841.887385] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4846.000076] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4847.706897] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:32:14 (1784050334) [ 5125.092283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5125.104285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.128334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.624949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.630486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.649814] Lustre: Skipped 1 previous similar message [ 5131.745272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.745272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.745281] Lustre: Skipped 2 previous similar messages [ 5131.753050] Lustre: Skipped 3 previous similar messages [ 5136.866970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5136.875895] Lustre: Skipped 2 previous similar messages [ 5137.377065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5137.717439] Lustre: server umount lustre-MDT0000 complete [ 5141.836964] LustreError: 160179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050630 with bad export cookie 3675876227370295892 [ 5141.837838] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5141.871741] LustreError: 160179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5141.984718] LustreError: 159380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5141.985501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5142.016968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5147.106430] LustreError: 163109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.201140] LustreError: 163124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.201317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5151.225181] Lustre: Skipped 1 previous similar message [ 5156.321549] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5156.835672] Lustre: server umount lustre-MDT0001 complete [ 5167.561252] Lustre: server umount lustre-OST0000 complete [ 5179.451395] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5189.972149] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5233.988710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.233310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5234.273296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5234.402871] Lustre: lustre-MDT0000: new disk, initializing [ 5234.471962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.491351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5239.946109] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5252.895695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.062558] Lustre: 166013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5253.149095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5253.164814] Lustre: Skipped 1 previous similar message [ 5253.320113] Lustre: lustre-MDT0001: new disk, initializing [ 5253.443233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5253.489931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5253.504809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5259.489703] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5271.035796] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5277.770735] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5285.066677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5285.351469] Lustre: lustre-OST0000: new disk, initializing [ 5285.359541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5285.364720] Lustre: 167983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5285.448562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.192646] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5294.219178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5294.237682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5294.388523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5303.489683] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5308.584935] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5308.743129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5313.957676] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5314.211426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5320.162251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5320.171819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.179442] Lustre: Skipped 1 previous similar message [ 5320.182453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.186464] Lustre: Skipped 2 previous similar messages [ 5323.134919] Lustre: server umount lustre-OST0000 complete [ 5330.405103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5330.418875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.421462] LustreError: Skipped 1 previous similar message [ 5330.467102] Lustre: Skipped 1 previous similar message [ 5335.824518] Lustre: server umount lustre-MDT0000 complete [ 5339.732268] LustreError: 166007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784050828 with bad export cookie 3675876227370454722 [ 5339.733466] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.756260] LustreError: 166007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5340.024414] Lustre: server umount lustre-MDT0001 complete [ 5351.412860] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:40:37 (1784050837) [ 5367.299451] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5381.021308] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5424.988540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.195398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5425.221374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5425.296818] Lustre: lustre-MDT0000: new disk, initializing [ 5425.381035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.396336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5431.425340] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5443.806258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.042965] Lustre: 172741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5444.732263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5449.893540] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5460.687750] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5468.933961] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5477.382786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5477.731130] Lustre: lustre-OST0000: new disk, initializing [ 5477.733171] Lustre: Skipped 1 previous similar message [ 5477.740952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5477.748226] Lustre: Skipped 2 previous similar messages [ 5477.756868] Lustre: 174721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5484.802351] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5486.667325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5486.673503] Lustre: Skipped 1 previous similar message [ 5486.677581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5486.784582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5494.681240] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5502.434391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5502.445034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.447886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.447894] Lustre: Skipped 4 previous similar messages [ 5502.448540] LustreError: Skipped 1 previous similar message [ 5502.501649] Lustre: Skipped 1 previous similar message [ 5508.646980] Lustre: server umount lustre-OST0000 complete [ 5518.884830] Lustre: server umount lustre-MDT0000 complete [ 5522.918546] LustreError: 172748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.938483] LustreError: 172748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5523.159274] LustreError: 172735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051011 with bad export cookie 3675876227370455604 [ 5523.170154] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5523.603392] Lustre: server umount lustre-MDT0001 complete [ 5532.766735] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:43:39 (1784051019) [ 5549.758839] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 5560.330964] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5598.420952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.646947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5598.678103] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5598.735366] Lustre: lustre-MDT0000: new disk, initializing [ 5598.794091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.800973] Lustre: Skipped 2 previous similar messages [ 5598.814097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5603.855518] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5614.217836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.327249] Lustre: 179044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5614.701040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5619.267364] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5630.718757] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5638.062573] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5644.905590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.289727] Lustre: 181022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5651.908919] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5653.094872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5653.323280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5662.103061] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5673.959089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.965025] Lustre: Skipped 2 previous similar messages [ 5673.967490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.969447] Lustre: Skipped 7 previous similar messages [ 5673.970827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5673.979260] LustreError: Skipped 1 previous similar message [ 5683.168633] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5683.526315] Lustre: server umount lustre-OST0000 complete [ 5694.441132] LustreError: 180191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5694.592416] Lustre: server umount lustre-MDT0000 complete [ 5699.609063] LustreError: 179844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051188 with bad export cookie 3675876227370456689 [ 5699.620339] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.626913] LustreError: 179844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5700.026390] Lustre: server umount lustre-MDT0001 complete [ 5711.558942] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 5714.465699] Key type lgssc unregistered [ 5714.749784] LNet: 182890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5714.757052] LNetError: 182890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5714.779792] LNet: Removed LNI 192.168.203.151@tcp [ 5715.838904] Key type .llcrypt unregistered [ 5715.841169] Key type ._llcrypt unregistered [ 5744.396323] Key type ._llcrypt registered [ 5744.402843] Key type .llcrypt registered [ 5744.518539] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5745.284113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5745.309906] alg: No test for adler32 (adler32-zlib) [ 5746.495610] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 5746.905892] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 5748.568172] Key type lgssc registered [ 5749.903086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5758.622838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5758.636361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.243704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.943475] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5770.726873] LustreError: 184308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.765937] LustreError: 184308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5775.843609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5775.858227] LustreError: 184309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.174896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5780.419351] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5789.052719] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5796.843155] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5806.079566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5806.301261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.869161] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5820.559807] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5830.921718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5851.635150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5851.659863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.660574] LustreError: Skipped 1 previous similar message [ 5851.661225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5851.673859] Lustre: Skipped 1 previous similar message [ 5851.697801] Lustre: Skipped 1 previous similar message [ 5854.794411] Lustre: server umount lustre-OST0000 complete [ 5861.863567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5861.869816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5861.887389] Lustre: Skipped 1 previous similar message [ 5861.902151] Lustre: Skipped 1 previous similar message [ 5865.141699] Lustre: server umount lustre-MDT0000 complete [ 5866.984527] LustreError: 184304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.016568] LustreError: 184304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5868.784086] LustreError: 184288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051357 with bad export cookie 9513891741279604235 [ 5868.788822] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.801859] LustreError: 184288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5869.071931] Lustre: server umount lustre-MDT0001 complete [ 5880.471561] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 5883.179233] Key type lgssc unregistered [ 5883.434643] LNet: 187868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5883.441358] LNetError: 187868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5883.463988] LNet: Removed LNI 192.168.203.151@tcp [ 5884.372736] Key type .llcrypt unregistered [ 5884.376146] Key type ._llcrypt unregistered [ 5911.335355] Key type ._llcrypt registered [ 5911.337426] Key type .llcrypt registered [ 5911.482846] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 5912.590492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5912.737479] alg: No test for adler32 (adler32-zlib) [ 5913.855108] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 5914.166427] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 5915.832262] Key type lgssc registered [ 5917.077485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5924.829981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5924.866700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.330300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.194714] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5936.619782] LustreError: 189286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.629437] LustreError: 189286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5940.069835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.369752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5945.184881] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5953.635815] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5960.076578] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5967.575805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.176647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.253625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5975.218427] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 5984.004706] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5991.909679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5991.920695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.924947] Lustre: lustre-OST0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 5991.944921] Lustre: Skipped 1 previous similar message [ 5996.001843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.011111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.018654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.035865] Lustre: lustre-OST0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 5997.057290] Lustre: Skipped 1 previous similar message [ 5997.694882] Lustre: server umount lustre-OST0000 complete [ 6007.378282] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f69180a1800.ost_server_uuid 50 [ 6009.276950] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6023.649048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.650815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6023.667631] Lustre: Skipped 1 previous similar message [ 6023.674486] Lustre: Skipped 1 previous similar message [ 6028.782416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6029.805103] Lustre: server umount lustre-MDT0000 complete [ 6033.704659] LustreError: 190021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051522 with bad export cookie 4789249373894457663 [ 6033.723527] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6033.728366] LustreError: 190021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6034.240699] Lustre: server umount lustre-MDT0001 complete [ 6043.632497] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:52:10 (1784051530) [ 6050.900549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.228340] LustreError: 193199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.309971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.045623] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6056.417761] LustreError: 193200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.544820] LustreError: 193199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6064.121903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.507418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.844642] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6077.103809] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.956437] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6090.385435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.750457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.671971] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6096.894075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6105.263759] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.965483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.179415] Lustre: lustre-OST0001: new disk, initializing [ 6112.182400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6112.189622] Lustre: 196041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6112.276430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.484499] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6119.008908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6119.024988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6119.205260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6128.583661] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6132.705483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6132.719919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.757158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6132.765658] Lustre: Skipped 1 previous similar message [ 6137.560639] Lustre: server umount lustre-OST0000 complete [ 6139.352226] LustreError: 195021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.604966] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6143.781251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6147.712986] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6147.894917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6149.614488] LustreError: 196427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6149.632231] LustreError: 196427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6165.473660] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6165.490395] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6165.493771] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6165.496683] Lustre: Skipped 2 previous similar messages [ 6165.511404] Lustre: Skipped 4 previous similar messages [ 6168.616208] Lustre: server umount lustre-OST0001 complete [ 6175.713951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.766649] Lustre: server umount lustre-MDT0000 complete [ 6180.833402] LustreError: 193196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.868282] LustreError: 193196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6183.297390] LustreError: 193182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051672 with bad export cookie 4789249373894458734 [ 6183.305301] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6183.319096] LustreError: 193182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6183.836221] Lustre: server umount lustre-MDT0001 complete [ 6208.385069] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:54:55 (1784051695) [ 6214.753125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.969741] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6214.992507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6215.148664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.218600] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6228.338425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.396686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6228.437839] Lustre: 199482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6233.278423] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6240.130293] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6245.664408] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6252.330583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.481615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6257.787919] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6258.826914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6265.061583] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6271.739339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.842290] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6277.837027] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6278.716875] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6285.943280] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6289.378225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.385221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.395257] Lustre: Skipped 1 previous similar message [ 6289.407430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.409505] Lustre: Skipped 2 previous similar messages [ 6294.926564] Lustre: server umount lustre-OST0000 complete [ 6297.052088] LustreError: 201333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.957315] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.970459] LustreError: Skipped 2 previous similar messages [ 6316.758099] Lustre: server umount lustre-OST0001 complete [ 6324.209602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.230251] Lustre: Skipped 4 previous similar messages [ 6326.642227] Lustre: server umount lustre-MDT0000 complete [ 6330.977348] LustreError: 199473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051819 with bad export cookie 4789249373894460134 [ 6330.980681] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.991760] LustreError: 199473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6331.488712] Lustre: server umount lustre-MDT0001 complete [ 6356.767776] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:57:23 (1784051843) [ 6373.314188] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 6384.197508] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 6418.815884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.062775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6419.132664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6419.202728] Lustre: lustre-MDT0000: new disk, initializing [ 6419.313751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6419.317661] Lustre: Skipped 3 previous similar messages [ 6419.328926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6423.629329] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6434.059420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.185490] Lustre: 207577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6434.221136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6434.224952] Lustre: Skipped 1 previous similar message [ 6434.294550] Lustre: lustre-MDT0001: new disk, initializing [ 6434.440447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6434.462116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6438.441831] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6447.846124] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6452.954377] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6460.027304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6460.308761] Lustre: lustre-OST0000: new disk, initializing [ 6460.316762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6460.329394] Lustre: 209546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6466.663945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6466.675025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6466.810967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6467.773926] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6477.419500] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.347904] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6482.620239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6487.181685] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6487.369589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6492.642169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6492.643237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6492.676400] LustreError: Skipped 2 previous similar messages [ 6492.701905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6492.711957] Lustre: Skipped 10 previous similar messages [ 6495.692346] Lustre: server umount lustre-OST0000 complete [ 6505.664923] Lustre: server umount lustre-MDT0000 complete [ 6508.001099] LustreError: 208799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.021587] LustreError: 208799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6509.578810] LustreError: 207569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051998 with bad export cookie 4789249373894461926 [ 6509.590643] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6509.600764] LustreError: 207569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6509.890894] Lustre: server umount lustre-MDT0001 complete [ 6518.248862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.434596] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6531.924172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.758123] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6543.673843] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6549.105764] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6554.970643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.234605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.238877] Lustre: Skipped 4 previous similar messages [ 6562.535565] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6571.382706] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6577.054873] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6577.284639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6581.869576] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6582.093435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6584.289508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6584.305761] LustreError: Skipped 1 previous similar message [ 6590.326631] Lustre: server umount lustre-OST0000 complete [ 6597.390316] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6602.802572] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6633.030684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.596956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.611761] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.613796] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6639.650608] Lustre: Skipped 1 previous similar message [ 6640.860781] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6649.370701] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6651.036051] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6660.065335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6660.070221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.081385] LustreError: Skipped 1 previous similar message [ 6660.088185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6660.107674] Lustre: Skipped 6 previous similar messages [ 6660.125707] Lustre: Skipped 8 previous similar messages [ 6669.280286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6669.486359] Lustre: server umount lustre-OST0000 complete [ 6679.795422] Lustre: server umount lustre-MDT0000 complete [ 6680.566587] LustreError: 214597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.582705] LustreError: 214597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6683.797644] LustreError: 212500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052172 with bad export cookie 4789249373894462808 [ 6683.806587] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6683.809534] LustreError: 212500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6684.210744] Lustre: server umount lustre-MDT0001 complete [ 6693.629640] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:03:00 (1784052180) [ 6700.824552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.476843] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6714.777103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.545880] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6726.607241] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6731.458748] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6737.992676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6745.422714] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6752.209695] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6756.617737] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6756.834621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6760.355733] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6760.592775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6766.061080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.248474] Lustre: lustre-OST0001: new disk, initializing [ 6766.252511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6766.260320] Lustre: 221116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6772.357881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6772.370107] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6772.441831] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6772.512413] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6782.512028] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6787.085823] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6787.323215] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6791.516867] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6791.720425] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6798.314875] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6799.836815] Lustre: server umount lustre-OST0001 complete [ 6806.668800] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6806.903917] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6811.240743] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6811.525359] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6839.795767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6840.079871] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6840.083139] Lustre: Skipped 5 previous similar messages [ 6844.938777] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6845.421888] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6845.431017] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6845.431231] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6845.445549] Lustre: Skipped 1 previous similar message [ 6851.453862] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6853.228632] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6861.481127] Lustre: server umount lustre-OST0001 complete [ 6884.365239] Lustre: server umount lustre-OST0000 complete [ 6893.715459] Lustre: server umount lustre-MDT0000 complete [ 6896.684330] LustreError: 217724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052385 with bad export cookie 4789249373894463774 [ 6896.687450] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6896.692052] LustreError: 217724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6903.090306] Lustre: server umount lustre-MDT0001 complete [ 6922.902363] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:06:50 (1784052410) [ 6928.526777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.757556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6928.782109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6932.695518] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6939.871241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.940564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6939.960955] Lustre: 226357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6943.860397] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6949.975288] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6954.511792] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6960.222961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.452281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6965.825530] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6971.231585] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6979.430122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6979.532588] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6985.763104] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 6992.959872] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6994.569663] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6998.571994] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6998.743496] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7002.680518] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7002.851608] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7007.507907] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f6906b96800.ost_server_uuid 50 [ 7008.726601] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f6906b96800.ost_server_uuid in IDLE state after 0 sec [ 7009.602502] Lustre: Permanently deactivating lustre-OST0001 [ 7009.616966] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7012.035786] Lustre: Permanently reactivating lustre-OST0001 [ 7016.426189] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7016.439365] Lustre: Skipped 10 previous similar messages [ 7016.443535] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7016.447844] Lustre: Skipped 17 previous similar messages [ 7021.134538] Lustre: server umount lustre-OST0001 complete [ 7021.541667] LustreError: 228203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.563561] LustreError: 228203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 7030.289807] Lustre: server umount lustre-OST0000 complete [ 7039.919214] Lustre: server umount lustre-MDT0000 complete [ 7043.339516] LustreError: 230765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052532 with bad export cookie 4789249373894465062 [ 7043.341800] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7043.351718] LustreError: 230765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7043.689804] Lustre: server umount lustre-MDT0001 complete [ 7068.602235] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:09:15 (1784052555) [ 7077.776899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.063359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7078.086615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7078.093476] Lustre: Skipped 5 previous similar messages [ 7081.581456] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7090.072045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.176902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7090.199486] Lustre: 233049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7093.644392] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7100.490217] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7105.898153] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7112.040798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7112.228258] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7112.249481] Lustre: Permanently deactivating lustre-OST0000 [ 7112.262815] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7118.411043] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7120.412600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7125.673777] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7131.962113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.080567] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7138.822514] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7139.551040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7146.721792] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7148.819511] Lustre: Permanently reactivating lustre-OST0000 [ 7148.842293] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7148.848049] Lustre: Skipped 2 previous similar messages [ 7155.804812] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7155.805318] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000d030c050, cur 1784052644 deadline 1784052639 last 1784052609 [ 7155.812696] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7155.834204] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7155.885376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7160.295683] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7160.305920] LustreError: Skipped 9 previous similar messages [ 7162.112592] Lustre: server umount lustre-OST0001 complete [ 7172.539596] Lustre: server umount lustre-OST0000 complete [ 7183.149388] Lustre: server umount lustre-MDT0000 complete [ 7187.027607] LustreError: 233039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052675 with bad export cookie 4789249373894466707 [ 7187.034220] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7187.045059] LustreError: 233039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7187.332517] Lustre: server umount lustre-MDT0001 complete [ 7199.417406] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7202.287705] Key type lgssc unregistered [ 7202.548791] LNet: 237904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7202.554622] LNetError: 237904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7202.570906] LNet: Removed LNI 192.168.203.151@tcp [ 7203.314114] Key type .llcrypt unregistered [ 7203.317809] Key type ._llcrypt unregistered [ 7224.979841] Key type ._llcrypt registered [ 7224.985212] Key type .llcrypt registered [ 7225.110948] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7226.283855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7226.328793] alg: No test for adler32 (adler32-zlib) [ 7227.435498] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 7227.698971] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7229.384258] Key type lgssc registered [ 7230.389781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7239.843239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7239.890321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.339805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7245.488820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7251.953159] LustreError: 239374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.987938] LustreError: 239374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7254.796965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7254.991396] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7255.041334] Lustre: 239363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7255.060621] Lustre: Permanently deactivating lustre-MDT0001 [ 7255.091905] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7255.629151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7257.440310] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7257.451767] LustreError: 239404:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7257.460916] LustreError: 239404:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7257.489045] LustreError: 239404:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7260.829267] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7268.322849] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7274.043231] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7281.133105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7281.498242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.633497] LustreError: 241216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7286.652751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7288.093820] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7291.880950] LustreError: 241215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.622338] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7297.001834] LustreError: 241217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.736668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.922325] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7302.970156] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7302.975031] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7307.199136] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7312.961202] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7316.540161] Lustre: Permanently reactivating lustre-MDT0001 [ 7316.554977] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7316.562453] Lustre: Skipped 2 previous similar messages [ 7321.074536] Lustre: lustre-MDT0001: Client 94feb607-5b0a-42cc-a444-e0d4403f9bec (at 192.168.203.51@tcp) reconnecting [ 7326.644063] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7328.164372] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7333.768147] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7333.971862] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7338.686460] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7338.861869] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7341.427772] Lustre: Permanently deactivating lustre-MDT0001 [ 7341.442384] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7341.447417] Lustre: Skipped 2 previous similar messages [ 7346.757492] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7352.806836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7352.817171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7352.828572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7355.365025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.368354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7355.378565] Lustre: Skipped 2 previous similar messages [ 7355.390624] Lustre: Skipped 2 previous similar messages [ 7359.017265] Lustre: server umount lustre-MDT0000 complete [ 7360.483905] LustreError: 243928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7360.499378] LustreError: 243928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7362.066160] LustreError: 239354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052850 with bad export cookie 14525420543447109607 [ 7362.067727] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7362.079514] LustreError: 239354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7362.334173] Lustre: server umount lustre-MDT0001 complete [ 7371.245509] Lustre: server umount lustre-OST0000 complete [ 7381.372648] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:14:28 (1784052868) [ 7391.213912] Lustre: server umount lustre-OST0001 complete [ 7397.168533] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 7406.537620] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7448.404740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.660312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7448.673385] Lustre: Skipped 2 previous similar messages [ 7448.711417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7448.859996] Lustre: lustre-MDT0000: new disk, initializing [ 7448.948241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.959925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7453.891433] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7465.321918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.511244] Lustre: 247632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7465.604459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7465.610650] Lustre: Skipped 1 previous similar message [ 7465.704514] Lustre: lustre-MDT0001: new disk, initializing [ 7465.861241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7465.932618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7465.963830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7470.555326] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7482.584887] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.039433] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7499.660030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7500.033551] Lustre: lustre-OST0000: new disk, initializing [ 7500.035964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7500.055652] Lustre: 249602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7500.248215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.893442] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7508.569457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7508.573175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7508.776052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7513.539493] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7515.573369] LustreError: 247638:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7523.983778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.275756] Lustre: lustre-OST0001: new disk, initializing [ 7524.284381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7524.297496] Lustre: 250721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7525.672072] LustreError: 247638:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7525.699098] LustreError: 247640:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7531.641594] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7533.130409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7533.136554] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7533.238135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7535.792458] LustreError: 247640:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7541.493538] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7543.226651] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7548.898370] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7548.913324] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.921626] LustreError: Skipped 1 previous similar message [ 7548.922595] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7548.936447] Lustre: Skipped 1 previous similar message [ 7548.940211] Lustre: Skipped 1 previous similar message [ 7551.601223] Lustre: server umount lustre-OST0001 complete [ 7554.020605] LustreError: 249585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7554.041015] LustreError: 249585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7569.379497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.411344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7569.420339] Lustre: Skipped 2 previous similar messages [ 7574.499515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7574.502548] LustreError: 249586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7574.509856] Lustre: Skipped 1 previous similar message [ 7574.535124] LustreError: 249586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7574.714040] Lustre: server umount lustre-OST0000 complete [ 7578.596674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7578.603422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.617511] Lustre: Skipped 1 previous similar message [ 7584.680808] Lustre: server umount lustre-MDT0000 complete [ 7588.952945] LustreError: 247624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053077 with bad export cookie 14525420543447112197 [ 7588.955115] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7588.965824] LustreError: 247624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7595.465128] Lustre: server umount lustre-MDT0001 complete [ 7607.142377] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7609.941666] Key type lgssc unregistered [ 7610.271610] LNet: 252757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7610.286718] LNetError: 252757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7610.324628] LNet: Removed LNI 192.168.203.151@tcp [ 7611.426087] Key type .llcrypt unregistered [ 7611.427944] Key type ._llcrypt unregistered [ 7654.447623] Key type ._llcrypt registered [ 7654.449611] Key type .llcrypt registered [ 7654.577644] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7656.075504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7656.132625] alg: No test for adler32 (adler32-zlib) [ 7657.300888] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 7657.584117] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7659.288243] Key type lgssc registered [ 7660.597644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7668.480536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7668.510591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.865433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7669.890175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7670.039462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.321385] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7682.681038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.789423] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7682.814467] Lustre: 254968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7683.027429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7687.707556] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7695.981328] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7702.052369] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7708.859302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.990108] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7709.130606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7715.114357] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7718.068552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7723.160406] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7779.299669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7779.324299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7779.349961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7781.341843] Lustre: server umount lustre-OST0000 complete [ 7786.539853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7802.035497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7802.665270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7809.245978] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7817.684451] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7820.015358] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7820.028633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7820.040151] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7820.051169] Lustre: Skipped 1 previous similar message [ 7861.216923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7861.221919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.226919] LustreError: Skipped 1 previous similar message [ 7861.236332] Lustre: Skipped 1 previous similar message [ 7861.240020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7861.242531] Lustre: Skipped 2 previous similar messages [ 7864.053741] Lustre: server umount lustre-OST0000 complete [ 7871.459647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.471979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7871.476474] Lustre: Skipped 1 previous similar message [ 7876.731571] Lustre: server umount lustre-MDT0000 complete [ 7881.698248] LustreError: 254980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7881.743416] LustreError: 254980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7883.141621] LustreError: 255719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053371 with bad export cookie 18279335847581836926 [ 7883.152777] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.155023] LustreError: 255719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7883.737535] Lustre: server umount lustre-MDT0001 complete [ 7898.060256] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 7901.201906] Key type lgssc unregistered [ 7901.621641] LNet: 260392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7901.640223] LNetError: 260392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7901.684356] LNet: Removed LNI 192.168.203.151@tcp [ 7902.769508] Key type .llcrypt unregistered [ 7902.773209] Key type ._llcrypt unregistered [ 7930.761939] Key type ._llcrypt registered [ 7930.763885] Key type .llcrypt registered [ 7930.851448] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 7932.136230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7932.223132] alg: No test for adler32 (adler32-zlib) [ 7933.301833] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 7933.522154] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 7935.179545] Key type lgssc registered [ 7936.153312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.495891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.512787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.905268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.862434] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7954.409205] LustreError: 261808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7954.428098] LustreError: 261808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7959.166462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.593214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7964.311097] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 7972.779360] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7979.572515] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7986.595819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.062076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7990.201645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7994.104805] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8001.825848] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8030.182982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.184857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.210499] Lustre: Skipped 1 previous similar message [ 8030.226045] Lustre: Skipped 1 previous similar message [ 8035.297967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8035.308658] Lustre: Skipped 1 previous similar message [ 8036.481320] Lustre: server umount lustre-OST0000 complete [ 8041.453518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8041.471438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.507335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.547793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.556157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8047.355514] Lustre: server umount lustre-MDT0000 complete [ 8050.660217] LustreError: 261807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.672968] LustreError: 261807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8052.164908] LustreError: 261789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053540 with bad export cookie 16495552502445993986 [ 8052.198078] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.802251] Lustre: server umount lustre-MDT0001 complete [ 8068.355809] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8071.869757] Key type lgssc unregistered [ 8072.172794] LNet: 266117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8072.181882] LNetError: 266117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8073.255444] LNet: Removed LNI 192.168.203.151@tcp [ 8074.018209] Key type .llcrypt unregistered [ 8074.020653] Key type ._llcrypt unregistered [ 8090.928525] Key type ._llcrypt registered [ 8090.931980] Key type .llcrypt registered [ 8091.099345] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8092.092548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8092.138947] alg: No test for adler32 (adler32-zlib) [ 8093.213757] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 8093.417777] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8095.097302] Key type lgssc registered [ 8096.500996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8104.287020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8104.326519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.941212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8110.599389] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8115.174853] LustreError: 267338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8115.194272] LustreError: 267338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8119.966816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.444860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8125.341724] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8134.378927] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8141.157361] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8148.637580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8149.011813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.095581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8156.432904] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8165.285234] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8176.611375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8176.627389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.643327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8180.706536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8180.717304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8180.731472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.944373] Lustre: server umount lustre-OST0000 complete [ 8190.951921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.955255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.963570] Lustre: Skipped 1 previous similar message [ 8190.985190] Lustre: Skipped 2 previous similar messages [ 8193.870872] Lustre: server umount lustre-MDT0000 complete [ 8196.066285] LustreError: 267338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8196.099258] LustreError: 267338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8198.166351] LustreError: 267320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053686 with bad export cookie 14327055166742791141 [ 8198.172491] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8198.189794] LustreError: 267320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8198.887594] Lustre: server umount lustre-MDT0001 complete [ 8210.822300] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8213.858893] Key type lgssc unregistered [ 8214.211371] LNet: 271014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8214.230936] LNetError: 271014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8214.256271] LNet: Removed LNI 192.168.203.151@tcp [ 8215.257794] Key type .llcrypt unregistered [ 8215.264885] Key type ._llcrypt unregistered [ 8240.540528] Key type ._llcrypt registered [ 8240.542558] Key type .llcrypt registered [ 8240.612939] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8242.013279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8242.069612] alg: No test for adler32 (adler32-zlib) [ 8243.085432] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 8243.417862] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8245.056191] Key type lgssc registered [ 8246.108101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8254.210708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8254.232802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.715552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.917887] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8265.201220] LustreError: 272437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8265.222888] LustreError: 272437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8268.962862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.432651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8274.668844] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8282.041822] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8287.422320] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8294.013648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8294.313840] LustreError: 274259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.502908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.527943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8302.403556] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8311.279289] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8340.453630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8340.464580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.479944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8341.478772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8341.479727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8341.479932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8342.814663] Lustre: server umount lustre-OST0000 complete [ 8351.201437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8351.211190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.228638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8351.236899] Lustre: Skipped 1 previous similar message [ 8353.987123] Lustre: server umount lustre-MDT0000 complete [ 8356.840104] LustreError: 272434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.858358] LustreError: 272434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8357.699909] LustreError: 272419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053846 with bad export cookie 13059439222884041425 [ 8357.718338] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8357.719244] LustreError: 272419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8358.034688] Lustre: server umount lustre-MDT0001 complete [ 8370.398529] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8373.036423] Key type lgssc unregistered [ 8373.385747] LNet: 276745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8373.400570] LNetError: 276745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8373.430357] LNet: Removed LNI 192.168.203.151@tcp [ 8374.004614] Key type .llcrypt unregistered [ 8374.006346] Key type ._llcrypt unregistered [ 8391.683635] Key type ._llcrypt registered [ 8391.689608] Key type .llcrypt registered [ 8391.821323] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8392.871086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.053982] alg: No test for adler32 (adler32-zlib) [ 8394.119481] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 8394.419552] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8396.185781] Key type lgssc registered [ 8398.001724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8407.570739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8407.589756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.029540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8413.695494] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8418.278874] LustreError: 277982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.296725] LustreError: 277982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8422.748513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.028976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.418712] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8434.879397] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8441.568845] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8449.465499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.870839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.065029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8456.589325] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8465.178491] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8482.788653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.801740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.807631] Lustre: Skipped 1 previous similar message [ 8486.689893] Lustre: server umount lustre-OST0000 complete [ 8493.027588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.032215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8493.042780] Lustre: Skipped 1 previous similar message [ 8493.045369] Lustre: Skipped 1 previous similar message [ 8496.719250] Lustre: server umount lustre-MDT0000 complete [ 8498.146809] LustreError: 277992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.168345] LustreError: 277992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8500.180162] LustreError: 277948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053989 with bad export cookie 3785805215741337329 [ 8500.182778] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8500.199733] LustreError: 277948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8500.714468] Lustre: server umount lustre-MDT0001 complete [ 8512.640452] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8515.842434] Key type lgssc unregistered [ 8516.176812] LNet: 281684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8516.189148] LNetError: 281684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8516.210361] LNet: Removed LNI 192.168.203.151@tcp [ 8516.980268] Key type .llcrypt unregistered [ 8516.983068] Key type ._llcrypt unregistered [ 8529.307400] Key type ._llcrypt registered [ 8529.309413] Key type .llcrypt registered [ 8529.465938] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8546.307833] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8563.423821] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8564.776457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8564.800943] alg: No test for adler32 (adler32-zlib) [ 8565.847964] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 8566.075280] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8567.736187] Key type lgssc registered [ 8568.875910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8604.442870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8604.488214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.855381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8605.942512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8606.090298] Lustre: lustre-MDT0000: new disk, initializing [ 8606.249571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.269904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8611.538507] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8623.803460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8623.972597] Lustre: 285238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8624.023753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8624.027782] Lustre: Skipped 1 previous similar message [ 8624.095242] Lustre: lustre-MDT0001: new disk, initializing [ 8624.224744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8624.288235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8624.300868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8629.450950] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8641.006153] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8647.947779] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8655.402060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.823367] Lustre: lustre-OST0000: new disk, initializing [ 8655.826692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8655.835465] Lustre: 287210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8655.923415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.890345] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8665.666595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8665.675579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8665.770392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8670.851171] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8675.271491] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8675.489558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8679.589787] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8679.842338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8686.565292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8686.570388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8686.575864] LustreError: Skipped 1 previous similar message [ 8686.577194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8686.590947] Lustre: Skipped 1 previous similar message [ 8691.699403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8691.706254] Lustre: Skipped 2 previous similar messages [ 8696.288215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8696.444300] Lustre: server umount lustre-OST0000 complete [ 8700.898211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8700.920119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.937248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8701.936065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.442997] Lustre: server umount lustre-MDT0000 complete [ 8710.095505] LustreError: 286043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054198 with bad export cookie 6490231142226392635 [ 8710.101072] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8710.109194] LustreError: 286043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8710.454536] Lustre: server umount lustre-MDT0001 complete [ 8717.701403] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:36:45 (1784054205) [ 8723.697952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.966755] LustreError: 289620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8723.976134] LustreError: 289620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8724.028240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.076632] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8729.064744] LustreError: 289621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8729.092902] LustreError: 289621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8734.181130] LustreError: 289620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8735.887124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.175613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8739.827374] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8746.464702] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8751.662396] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.825534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.074131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.040750] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8771.155517] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8784.871825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8784.879424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.906129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.907626] Lustre: Skipped 1 previous similar message [ 8784.913994] Lustre: Skipped 3 previous similar messages [ 8789.546402] Lustre: server umount lustre-OST0000 complete [ 8795.106256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.132489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8795.148165] Lustre: Skipped 1 previous similar message [ 8795.179250] Lustre: Skipped 1 previous similar message [ 8800.534500] Lustre: server umount lustre-MDT0000 complete [ 8803.961070] LustreError: 289601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054292 with bad export cookie 6490231142226393517 [ 8803.970732] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8803.972497] LustreError: 289601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8804.242849] Lustre: server umount lustre-MDT0001 complete [ 8813.685368] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 8816.331937] Key type lgssc unregistered [ 8816.622401] LNet: 293151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8816.632326] LNetError: 293151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8816.650809] LNet: Removed LNI 192.168.203.151@tcp [ 8817.550745] Key type .llcrypt unregistered [ 8817.553051] Key type ._llcrypt unregistered [ 8845.784827] Key type ._llcrypt registered [ 8845.793421] Key type .llcrypt registered [ 8846.015821] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 8861.802688] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 8862.877620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8863.016610] alg: No test for adler32 (adler32-zlib) [ 8864.061512] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 8864.327959] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 8866.000242] Key type lgssc registered [ 8867.244689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8914.464944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8914.514719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.713715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8915.746964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8915.848612] Lustre: lustre-MDT0000: new disk, initializing [ 8915.956634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.970151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8920.122582] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8931.046558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.177767] Lustre: 296980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8931.232748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8931.232762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8931.237877] Lustre: Skipped 1 previous similar message [ 8931.303633] Lustre: lustre-MDT0001: new disk, initializing [ 8931.449420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8931.493000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8931.507578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8937.453071] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8948.831693] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8955.080566] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8962.535542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8962.849726] Lustre: lustre-OST2710: new disk, initializing [ 8962.854342] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8962.859224] Lustre: 298948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8962.909812] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8968.490176] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8970.944280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8970.953957] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8971.054507] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8977.887948] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8984.780773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8984.935506] Lustre: lustre-OST03e8: new disk, initializing [ 8984.938969] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8984.944734] Lustre: 300016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8985.035407] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8990.989984] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 8991.811737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8991.828599] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8991.907153] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8998.268558] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9006.090380] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9006.352906] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9010.857419] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9011.116114] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9015.884637] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9016.148058] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9020.521858] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9020.725253] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9028.073612] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9028.083101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9028.097582] Lustre: Skipped 1 previous similar message [ 9028.112832] Lustre: Skipped 3 previous similar messages [ 9031.133908] Lustre: server umount lustre-MDT0000 complete [ 9034.742221] LustreError: 296973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054523 with bad export cookie 16047021814242593918 [ 9034.748863] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9034.755699] LustreError: 296973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9035.131924] Lustre: server umount lustre-MDT0001 complete [ 9049.260054] Lustre: server umount lustre-OST2710 complete [ 9062.554994] Lustre: server umount lustre-OST03e8 complete [ 9081.683997] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [ 9094.096904] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9141.545447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.933526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9142.051948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9142.175048] Lustre: lustre-MDT0000: new disk, initializing [ 9142.306775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.332866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9146.637118] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9158.302133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.381421] Lustre: 305269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9158.424174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9158.429193] Lustre: Skipped 1 previous similar message [ 9158.491065] Lustre: lustre-MDT0001: new disk, initializing [ 9158.589659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9158.633102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9158.655996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9163.296492] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9173.328802] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9178.737388] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9185.730805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.951322] Lustre: lustre-OST0000: new disk, initializing [ 9185.954797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9185.960625] Lustre: 307239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9186.057475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.193971] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9196.104963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9196.127661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9196.287889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9202.264657] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9207.556501] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9207.776819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9213.266774] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9213.593567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9216.994107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9217.002971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.019937] LustreError: Skipped 1 previous similar message [ 9217.021223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9217.079495] Lustre: Skipped 3 previous similar messages [ 9222.118396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.131592] Lustre: Skipped 1 previous similar message [ 9227.236318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9227.239439] Lustre: Skipped 1 previous similar message [ 9230.304717] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9230.461740] Lustre: server umount lustre-OST0000 complete [ 9237.476917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.486617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9237.503871] Lustre: Skipped 1 previous similar message [ 9237.520619] Lustre: Skipped 2 previous similar messages [ 9241.959312] Lustre: server umount lustre-MDT0000 complete [ 9242.595964] LustreError: 305281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9242.619749] LustreError: 305281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9245.834261] LustreError: 305262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054734 with bad export cookie 16047021814242595598 [ 9245.843136] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9245.856940] LustreError: 305262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9246.186300] Lustre: server umount lustre-MDT0001 complete [ 9257.358512] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:45:44 (1784054744) [ 9258.709142] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9260.952736] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:45:47 (1784054747) [ 9264.897631] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9298.218941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.481732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9298.513666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9298.690178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.091459] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9313.774438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.942870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9313.964169] Lustre: 311098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9319.733910] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9343.218778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9343.444985] Lustre: Denying initial registration attempt from nid 192.168.203.151@tcp, specified as failover [ 9343.457282] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9343.471980] LustreError: 312554:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9343.487797] LustreError: 312554:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9343.505959] LustreError: 312554:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9343.589843] Lustre: server umount lustre-OST0000 complete [ 9343.595840] LustreError: 312554:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9350.120350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9350.130412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.142091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9356.545144] Lustre: server umount lustre-MDT0000 complete [ 9356.770929] LustreError: 311924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.809030] LustreError: 311924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9360.411179] LustreError: 311091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054849 with bad export cookie 16047021814242596480 [ 9360.425047] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9360.431296] LustreError: 311091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9360.966824] Lustre: server umount lustre-MDT0001 complete [ 9372.240972] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9375.604702] Key type lgssc unregistered [ 9375.910956] LNet: 313674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9375.916687] LNetError: 313674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9375.932043] LNet: Removed LNI 192.168.203.151@tcp [ 9376.859600] Key type .llcrypt unregistered [ 9376.863704] Key type ._llcrypt unregistered [ 9389.590615] Key type ._llcrypt registered [ 9389.592475] Key type .llcrypt registered [ 9389.742895] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9390.884955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9391.123067] alg: No test for adler32 (adler32-zlib) [ 9392.190334] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 9392.440248] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9394.104171] Key type lgssc registered [ 9395.246625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9442.856478] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9451.583300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9451.625362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.248569] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9453.311199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9453.649873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9458.085460] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9465.808720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.917816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9465.982103] Lustre: 316437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9466.221295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9470.089994] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9479.327545] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9486.218766] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9508.644783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.859971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9509.064559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9515.038864] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9524.389585] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9527.786240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9527.801870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9527.804977] Lustre: Skipped 1 previous similar message [ 9532.529524] Lustre: server umount lustre-OST0000 complete [ 9538.017356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9538.023373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.041485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9538.064936] Lustre: Skipped 2 previous similar messages [ 9542.893438] Lustre: server umount lustre-MDT0000 complete [ 9547.491883] LustreError: 317189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055036 with bad export cookie 17974092057508378573 [ 9547.511634] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.953377] Lustre: server umount lustre-MDT0001 complete [ 9565.927228] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9570.841969] Key type lgssc unregistered [ 9571.452861] LNet: 320036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9571.467242] LNetError: 320036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9572.520257] LNet: Removed LNI 192.168.203.151@tcp [ 9573.702429] Key type .llcrypt unregistered [ 9573.709493] Key type ._llcrypt unregistered [ 9599.508608] Key type ._llcrypt registered [ 9599.511458] Key type .llcrypt registered [ 9599.608394] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9600.632740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9600.721644] alg: No test for adler32 (adler32-zlib) [ 9601.736533] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 9601.975546] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9603.656226] Key type lgssc registered [ 9604.810715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9612.001624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9612.018623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9613.382858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.875679] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9623.523420] LustreError: 321455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.544560] LustreError: 321455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9625.584148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.021891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9631.090547] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9639.834922] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9646.914802] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9657.333974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.757530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.887900] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9673.474138] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9682.401696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9682.405052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.414347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9686.002812] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [ 9686.514677] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9686.531769] Lustre: Skipped 1 previous similar message [ 9687.974448] Lustre: server umount lustre-MDT0000 complete [ 9691.120099] LustreError: 321452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.126817] LustreError: 323300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055180 with bad export cookie 15860053668120501621 [ 9692.129205] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9692.137915] LustreError: 323300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9696.214647] LustreError: 321451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9696.248937] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [ 9696.259916] LustreError: 321451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9696.301981] Lustre: Skipped 3 previous similar messages [ 9696.740711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.588785] Lustre: server umount lustre-MDT0001 complete [ 9705.889412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9715.056545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9717.216131] LustreError: 325035:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9717.216997] LustreError: 320764:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a02fd14c x1870717350093824/t0(0) o250->MGC192.168.203.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9717.565559] LustreError: 323288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9717.589662] LustreError: 323288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9717.639980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.724314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9722.490572] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9722.731891] LustreError: 325048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.752027] LustreError: 325048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9727.860839] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9730.901491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.183477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9735.481420] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9736.695861] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9743.237211] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9745.068465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9751.046152] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9752.411542] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9757.159121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.163462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9757.173796] Lustre: Skipped 1 previous similar message [ 9757.192456] Lustre: Skipped 3 previous similar messages [ 9762.275663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9762.287063] Lustre: Skipped 1 previous similar message [ 9763.377887] Lustre: server umount lustre-OST0000 complete [ 9772.001743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9772.015188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9772.037198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9773.873830] Lustre: server umount lustre-MDT0000 complete [ 9777.635259] LustreError: 325053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.649335] LustreError: 325053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9778.293914] LustreError: 321435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055267 with bad export cookie 15860053668120509174 [ 9778.302276] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9778.314623] LustreError: 321435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9778.799505] Lustre: server umount lustre-MDT0001 complete [ 9790.394175] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [ 9793.338382] Key type lgssc unregistered [ 9793.696751] LNet: 327720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9793.705991] LNetError: 327720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9794.741112] LNet: Removed LNI 192.168.203.151@tcp [ 9796.202710] Key type .llcrypt unregistered [ 9796.205342] Key type ._llcrypt unregistered [ 9827.122965] Key type ._llcrypt registered [ 9827.128736] Key type .llcrypt registered [ 9827.229062] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [ 9828.111623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9828.135636] alg: No test for adler32 (adler32-zlib) [ 9829.162553] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [ 9829.398650] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [ 9831.096805] Key type lgssc registered [ 9832.211337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9839.390539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9839.402348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9841.003536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9845.503210] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9850.340380] LustreError: 329137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9850.352695] LustreError: 329137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9854.013660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9854.508416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9858.736835] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9867.511736] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9875.064935] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9879.974481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.999244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9880.008653] Lustre: Skipped 1 previous similar message [ 9885.156470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9890.277810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9890.290419] Lustre: Skipped 2 previous similar messages [ 9893.347097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9893.690932] Lustre: server umount lustre-MDT0000 complete [ 9895.394184] LustreError: 329133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9895.420136] LustreError: 329133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9897.620150] LustreError: 329874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055386 with bad export cookie 12773920768337218341 [ 9897.629924] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.641889] LustreError: 329874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9898.121422] Lustre: server umount lustre-MDT0001 complete [ 9915.345511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.681627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9915.746133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9916.029695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9921.100052] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9930.478801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9930.631161] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9930.661659] Lustre: 331729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9930.994845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9935.781703] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9944.139215] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9951.244956] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9961.387848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9961.600646] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9961.614472] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9961.621856] LustreError: 333613:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9961.631968] LustreError: 333613:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9961.641771] LustreError: 333613:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9961.744635] Lustre: server umount lustre-OST0000 complete [ 9961.746697] LustreError: 333613:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9973.997448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9974.247346] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9974.554060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9982.278735] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [ 9984.176431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9990.976776] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9999.336894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9999.433583] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9999.511821] Lustre: lustre-OST0001: new disk, initializing [ 9999.516586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9999.523765] Lustre: 335154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9999.620929] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10005.148280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10005.162432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10005.301148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10006.729664] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10015.781710] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10020.833662] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10020.849252] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.870818] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10020.872078] Lustre: Skipped 2 previous similar messages [10020.890712] Lustre: Skipped 1 previous similar message [10025.954057] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10025.959796] Lustre: Skipped 1 previous similar message [10033.120386] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10033.178241] Lustre: server umount lustre-OST0001 complete [10036.204262] LustreError: 334079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.222276] LustreError: 334079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10040.290151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10040.299705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.319509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10040.322638] Lustre: Skipped 2 previous similar messages [10041.331936] LustreError: 335452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10041.346733] LustreError: 335452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10043.333350] Lustre: server umount lustre-OST0000 complete [10048.993202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10049.004385] LustreError: Skipped 1 previous similar message [10049.011416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.031653] Lustre: Skipped 1 previous similar message [10053.330105] Lustre: server umount lustre-MDT0000 complete [10056.674072] LustreError: 331740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.704112] LustreError: 331740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10056.792915] LustreError: 331721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055545 with bad export cookie 12773920768337218894 [10056.807026] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10056.809305] LustreError: 331721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10057.196715] Lustre: server umount lustre-MDT0001 complete [10068.675987] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10071.318701] Key type lgssc unregistered [10071.662234] LNet: 337115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10071.673844] LNetError: 337115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10071.693839] LNet: Removed LNI 192.168.203.151@tcp [10072.468168] Key type .llcrypt unregistered [10072.471032] Key type ._llcrypt unregistered [10150.000293] Key type ._llcrypt registered [10150.001410] Key type .llcrypt registered [10150.056207] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [10170.852188] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10172.025500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10172.064115] alg: No test for adler32 (adler32-zlib) [10173.373339] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [10173.949446] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10175.786789] Key type lgssc registered [10177.534516] Lustre: Echo OBD driver; http://www.lustre.org/ [10216.783875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10216.849987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10218.152954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10218.191204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10218.299449] Lustre: lustre-MDT0000: new disk, initializing [10218.391866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10218.406907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10223.566704] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10235.398128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.595079] Lustre: 342065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10235.662031] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10235.666100] Lustre: Skipped 1 previous similar message [10235.771303] Lustre: lustre-MDT0001: new disk, initializing [10235.941312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10236.002636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10236.026443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10240.472078] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10249.779511] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10256.191628] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10264.360928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10264.633199] Lustre: lustre-OST0000: new disk, initializing [10264.638135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10264.653732] Lustre: 344035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10264.743169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10271.124437] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10272.864685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10272.881241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10272.995263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10281.237640] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10285.910996] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10286.184608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10290.461297] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10290.651639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10293.730287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10293.737822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10293.759275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10298.574949] Lustre: server umount lustre-OST0000 complete [10303.971909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10303.977464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10303.989044] Lustre: Skipped 2 previous similar messages [10303.996986] Lustre: Skipped 1 previous similar message [10309.090404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10309.103586] Lustre: Skipped 2 previous similar messages [10313.186148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10313.196875] Lustre: Skipped 1 previous similar message [10317.284118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10317.549452] Lustre: server umount lustre-MDT0000 complete [10322.260035] LustreError: 342057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055811 with bad export cookie 17133670190526070604 [10322.265054] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.288613] LustreError: 342057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10322.946672] Lustre: server umount lustre-MDT0001 complete [10334.992146] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:03:41 (1784055821) [10343.016576] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:03:49 (1784055829) [10352.013187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.510222] LustreError: 346792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10352.540916] LustreError: 346792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10352.613269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10356.783381] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10357.730866] LustreError: 346793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.849490] LustreError: 346792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10365.356315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.709078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10370.373531] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10379.340558] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10386.559175] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10393.833954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10394.219536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10400.786894] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10409.011597] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10420.420922] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10422.482477] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10424.028544] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10425.747425] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10427.645178] Lustre: Failing over lustre-MDT0000 [10427.945660] Lustre: server umount lustre-MDT0000 complete [10431.470609] LustreError: 346787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.484899] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.486306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10432.500664] Lustre: Skipped 1 previous similar message [10436.577316] LustreError: 346789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.598116] LustreError: 346789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10445.884853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10446.073777] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10446.348365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10446.403372] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10446.816506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10450.685308] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10451.442819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10451.469543] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10451.518533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10459.733657] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10461.423636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10465.017624] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10467.148364] Lustre: Failing over lustre-MDT0000 [10467.363554] LustreError: 346789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10467.374754] LustreError: 346789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10467.483376] Lustre: server umount lustre-MDT0000 complete [10471.916678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10471.924383] Lustre: Skipped 2 previous similar messages [10471.942312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10487.276167] LustreError: 347520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.308704] LustreError: 347520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [10487.777144] Lustre: 340288:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784055960/real 1784055960] req@0000000070ce079e x1870717949929088/t0(0) o400->MGC192.168.203.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784055976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10487.868218] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.531499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.062509] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xedc706429d0189e8 [10498.078521] LustreError: 351207:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.151@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000c89ba8da) refcount nonzero (1) after lock cleanup; forcing cleanup. [10498.112646] Lustre: MGC192.168.203.151@tcp: Connection restored to 0@lo (at 0@lo) [10498.123111] Lustre: Skipped 2 previous similar messages [10498.487347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10498.590798] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10502.637220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10502.792243] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10503.664540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10503.698271] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10503.738620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10511.562590] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10513.093453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10524.136220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.141784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10524.147765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.150511] Lustre: Skipped 2 previous similar messages [10525.375214] Lustre: server umount lustre-MDT0000 complete [10529.138616] LustreError: 346773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056017 with bad export cookie 17133670190526073320 [10529.149030] LustreError: 346773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10529.157707] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10529.249853] LustreError: 346787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.275231] LustreError: 346787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10529.577881] Lustre: server umount lustre-MDT0001 complete [10536.154261] Lustre: server umount lustre-OST0000 complete [10552.924845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10557.840712] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10567.374881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.817725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10567.821740] Lustre: Skipped 1 previous similar message [10573.504448] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10582.361258] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10588.657308] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10595.580984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10599.104906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10602.415227] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10610.260847] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10619.881484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10619.890742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10619.903440] Lustre: Skipped 2 previous similar messages [10619.908763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10619.912945] Lustre: Skipped 3 previous similar messages [10626.158770] Lustre: server umount lustre-OST0000 complete [10636.522413] Lustre: server umount lustre-MDT0000 complete [10638.305630] LustreError: 353152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.331416] LustreError: 353152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10640.419478] LustreError: 353134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056129 with bad export cookie 17133670190526074069 [10640.422330] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10640.434066] LustreError: 353134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10640.721586] Lustre: server umount lustre-MDT0001 complete [10651.052293] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [10653.383320] Key type lgssc unregistered [10653.614317] LNet: 356688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10653.623860] LNetError: 356688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10653.643021] LNet: Removed LNI 192.168.203.151@tcp [10654.689089] Key type .llcrypt unregistered [10654.692415] Key type ._llcrypt unregistered [10685.556764] Key type ._llcrypt registered [10685.561905] Key type .llcrypt registered [10685.667958] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [10686.906014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10687.023136] alg: No test for adler32 (adler32-zlib) [10688.225823] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [10688.564660] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [10690.272199] Key type lgssc registered [10691.387740] Lustre: Echo OBD driver; http://www.lustre.org/ [10698.470881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10698.483483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.954081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10704.243604] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10709.481927] LustreError: 358205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10709.506978] LustreError: 358205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10713.545213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.200215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10718.999772] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10726.667434] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10732.684326] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10739.909940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10740.312841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10744.445022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10746.656714] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10755.632349] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10774.501694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10774.525338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10774.537120] Lustre: Skipped 1 previous similar message [10776.802637] Lustre: server umount lustre-MDT0000 complete [10779.629748] LustreError: 358205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.648189] LustreError: 358205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10781.030391] LustreError: 358186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056269 with bad export cookie 2852978856293907559 [10781.039724] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10781.044564] LustreError: 358186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10781.581722] Lustre: server umount lustre-MDT0001 complete [10796.368872] Lustre: server umount lustre-OST0000 complete [10811.001283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10811.611054] LustreError: 361936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.723371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10816.550353] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10816.995165] LustreError: 361937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10822.114568] LustreError: 361936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10825.281791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10825.544180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10830.283083] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10838.225943] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10844.478541] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10851.252175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.630204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10856.752279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10858.592763] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10866.576616] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10881.511740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.523491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10881.529615] Lustre: Skipped 2 previous similar messages [10881.540596] Lustre: Skipped 2 previous similar messages [10884.977870] Lustre: server umount lustre-MDT0000 complete [10886.637280] LustreError: 361936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.664934] LustreError: 361936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10889.069474] LustreError: 361918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056377 with bad export cookie 2852978856293908672 [10889.076850] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10889.085674] LustreError: 361918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10889.416660] Lustre: server umount lustre-MDT0001 complete [10903.818805] Lustre: server umount lustre-OST0000 complete [10922.382706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.960127] LustreError: 365851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.056830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10928.524536] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10937.557191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.986320] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10953.266659] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10959.426757] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10967.140529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10967.555366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10967.558321] Lustre: Skipped 1 previous similar message [10968.623986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10973.707889] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [10981.077672] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10988.017520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10988.039093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.044155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10988.063115] Lustre: Skipped 2 previous similar messages [10988.082387] Lustre: Skipped 3 previous similar messages [10992.903456] Lustre: server umount lustre-OST0000 complete [10998.243432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.253085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10998.270395] Lustre: Skipped 1 previous similar message [10998.282104] Lustre: Skipped 1 previous similar message [11003.341166] Lustre: server umount lustre-MDT0000 complete [11003.362143] LustreError: 365852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.405704] LustreError: 365852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11007.139178] LustreError: 365833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056495 with bad export cookie 2852978856293909701 [11007.147770] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11007.153759] LustreError: 365833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11007.407762] Lustre: server umount lustre-MDT0001 complete [11018.987430] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11022.631425] Key type lgssc unregistered [11023.080805] LNet: 369383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11023.086341] LNetError: 369383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11023.100429] LNet: Removed LNI 192.168.203.151@tcp [11024.353533] Key type .llcrypt unregistered [11024.360210] Key type ._llcrypt unregistered [11051.561302] Key type ._llcrypt registered [11051.565422] Key type .llcrypt registered [11051.662385] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11052.636849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11052.670435] alg: No test for adler32 (adler32-zlib) [11053.592899] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [11053.769955] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11055.400188] Key type lgssc registered [11056.501693] Lustre: Echo OBD driver; http://www.lustre.org/ [11064.085066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11064.142684] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.161057] LustreError: 370826:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11064.201438] LustreError: 370826:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11064.212565] LustreError: 370826:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11064.231375] LustreError: 370826:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11064.261038] LustreError: 370826:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11072.875772] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11072.893881] LustreError: 371223:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11072.924895] LustreError: 371223:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11072.941880] LustreError: 371223:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11072.958677] LustreError: 371223:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11072.974702] LustreError: 371223:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11092.577249] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11095.052710] Key type lgssc unregistered [11095.327545] LNet: 372227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11095.334409] LNetError: 372227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11095.353242] LNet: Removed LNI 192.168.203.151@tcp [11096.147450] Key type .llcrypt unregistered [11096.153448] Key type ._llcrypt unregistered [11114.660690] Key type ._llcrypt registered [11114.664418] Key type .llcrypt registered [11114.762429] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11129.734960] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11130.415574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11130.474407] alg: No test for adler32 (adler32-zlib) [11131.545154] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [11131.795364] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11133.466548] Key type lgssc registered [11134.502629] Lustre: Echo OBD driver; http://www.lustre.org/ [11169.817031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11169.864699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.212557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11171.247244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11171.343842] Lustre: lustre-MDT0000: new disk, initializing [11171.436092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11171.468293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11176.127078] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11187.808804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11187.947615] Lustre: 375382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11188.048933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11188.053123] Lustre: Skipped 1 previous similar message [11188.129290] Lustre: lustre-MDT0001: new disk, initializing [11188.268963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11188.300048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11188.311062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11193.254581] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11203.229317] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11209.689977] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11217.020958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11217.239638] Lustre: lustre-OST0000: new disk, initializing [11217.245390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11217.258833] Lustre: 377352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11217.329089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11223.206645] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11223.643068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11223.651903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11223.783693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11233.328990] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11238.298752] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11238.484559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11243.491696] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11243.745718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11249.639495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11249.652697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.663408] LustreError: Skipped 1 previous similar message [11249.670119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11249.689675] Lustre: Skipped 1 previous similar message [11249.717615] Lustre: Skipped 1 previous similar message [11254.757689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11254.769226] Lustre: Skipped 1 previous similar message [11259.879703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11259.896238] Lustre: Skipped 1 previous similar message [11260.384788] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11260.530905] Lustre: server umount lustre-OST0000 complete [11264.993096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11264.995423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.019424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11265.038116] Lustre: Skipped 1 previous similar message [11265.060773] Lustre: Skipped 1 previous similar message [11270.115425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11270.123621] Lustre: Skipped 1 previous similar message [11270.943700] Lustre: server umount lustre-MDT0000 complete [11274.539737] LustreError: 375376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056763 with bad export cookie 12636133289072305865 [11274.553307] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.555975] LustreError: 375376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11274.933708] Lustre: server umount lustre-MDT0001 complete [11283.864413] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11286.203982] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:19:33 (1784056773) [11293.578043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.176366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11299.298663] LustreError: 379816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.318359] LustreError: 379816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11299.484856] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11303.401104] LustreError: 379815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.515214] LustreError: 379816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.051555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.502436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11315.235825] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11323.372857] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11329.378876] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11336.233765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11336.611965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11342.497504] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11349.282861] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11355.792929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11355.963228] Lustre: lustre-OST0001: new disk, initializing [11355.967403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11355.975608] Lustre: 382653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11356.116947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11361.880531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11361.899764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11361.976873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11362.697871] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11373.557738] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11377.639505] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11377.648105] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11377.663912] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11377.670807] Lustre: Skipped 1 previous similar message [11383.438538] Lustre: server umount lustre-OST0001 complete [11385.820492] LustreError: 381639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.002179] LustreError: 383326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.018875] LustreError: 383326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11402.209734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11402.217472] LustreError: Skipped 1 previous similar message [11402.231920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11402.250776] Lustre: Skipped 1 previous similar message [11402.258204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11402.267957] Lustre: Skipped 3 previous similar messages [11403.267989] LustreError: 382961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11403.294831] LustreError: 382961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11407.590826] Lustre: server umount lustre-OST0000 complete [11412.961313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11412.974474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11412.995661] Lustre: Skipped 1 previous similar message [11418.029980] Lustre: server umount lustre-MDT0000 complete [11422.417877] LustreError: 379797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056911 with bad export cookie 12636133289072306740 [11422.425785] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11422.431078] LustreError: 379797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11429.008528] Lustre: server umount lustre-MDT0001 complete [11441.701498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11445.097799] Key type lgssc unregistered [11445.518792] LNet: 384642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11445.530405] LNetError: 384642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11445.550955] LNet: Removed LNI 192.168.203.151@tcp [11446.675216] Key type .llcrypt unregistered [11446.681201] Key type ._llcrypt unregistered [11508.153913] Key type ._llcrypt registered [11508.159616] Key type .llcrypt registered [11508.293179] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11509.722833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11509.760395] alg: No test for adler32 (adler32-zlib) [11510.820729] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [11511.141588] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11512.864609] Key type lgssc registered [11514.516957] Lustre: Echo OBD driver; http://www.lustre.org/ [11523.510561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11523.550039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11524.828871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11524.845739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11524.986213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11530.085726] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11540.038909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.192576] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11540.258859] Lustre: 387444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11540.541150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11545.609153] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11553.681914] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11559.497749] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11563.493063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11563.496210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11563.515904] Lustre: Skipped 1 previous similar message [11567.747472] Lustre: server umount lustre-MDT0000 complete [11571.866406] LustreError: 387435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057060 with bad export cookie 15935774355366993902 [11571.868121] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11571.881467] LustreError: 387435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11572.445261] Lustre: server umount lustre-MDT0001 complete [11583.224804] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:24:30 (1784057070) [11591.308905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.904098] LustreError: 389958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11591.931147] LustreError: 389958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11591.962115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11596.631338] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11597.302113] LustreError: 389959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11602.402167] LustreError: 389958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11606.049224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.418262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11610.915270] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11620.949607] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11628.464617] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11636.396918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.596200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11636.799239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11643.829385] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11652.707936] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11661.677515] Lustre: Permanently deactivating lustre-OST0000 [11661.688653] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11669.018844] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11671.512758] LustreError: 392945:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11671.527394] LustreError: 392945:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11671.535521] LustreError: 392945:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11674.740687] LustreError: 392993:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11674.746504] LustreError: 392993:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11682.962334] Lustre: server umount lustre-OST0000 complete [11685.690587] LustreError: 393243:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11685.695337] LustreError: 393243:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11687.905648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.917294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.920793] Lustre: Skipped 1 previous similar message [11693.028662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.040117] Lustre: Skipped 1 previous similar message [11698.156734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11698.169830] Lustre: Skipped 1 previous similar message [11698.238695] Lustre: server umount lustre-MDT0000 complete [11701.963721] LustreError: 389939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057190 with bad export cookie 15935774355366994483 [11701.969806] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11701.973275] LustreError: 389939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11711.640624] Lustre: server umount lustre-MDT0001 complete [11720.547188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.416702] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11728.126658] LustreError: 394402:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11734.599413] Lustre: 394741:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.151@tcp [11736.945875] Lustre: server umount MGS complete [11749.606202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.047823] LustreError: 395466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11750.091880] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11750.095664] Lustre: Skipped 1 previous similar message [11750.142806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11755.003511] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11755.490023] LustreError: 395467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.861129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11764.094500] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11768.786160] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11777.220057] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11782.751121] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11784.540133] Lustre: Permanently reactivating lustre-OST0000 [11784.558632] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11784.563675] Lustre: Skipped 2 previous similar messages [11813.344279] LustreError: 397268:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11821.825046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11822.271035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11822.275409] Lustre: Skipped 1 previous similar message [11828.348899] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11835.406409] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11843.045741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11843.049879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.059906] Lustre: Skipped 1 previous similar message [11843.072822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11844.068225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11854.307541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.320747] Lustre: Skipped 5 previous similar messages [11855.328692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11855.438536] Lustre: server umount lustre-OST0000 complete [11859.425869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.441854] Lustre: Skipped 1 previous similar message [11865.097980] Lustre: server umount lustre-MDT0000 complete [11869.061147] LustreError: 395446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057357 with bad export cookie 15935774355366995533 [11869.074116] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11869.079432] LustreError: 395446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11869.615068] Lustre: server umount lustre-MDT0001 complete [11881.734143] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [11884.442880] Key type lgssc unregistered [11884.713663] LNet: 399881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11884.731032] LNetError: 399881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11884.760507] LNet: Removed LNI 192.168.203.151@tcp [11885.733495] Key type .llcrypt unregistered [11885.736153] Key type ._llcrypt unregistered [11905.521683] Key type ._llcrypt registered [11905.524830] Key type .llcrypt registered [11905.655192] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [11923.503988] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [11925.045752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11925.127071] alg: No test for adler32 (adler32-zlib) [11926.277502] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [11926.696491] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [11928.440177] Key type lgssc registered [11929.659649] Lustre: Echo OBD driver; http://www.lustre.org/ [11965.282514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11965.329240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.645426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11966.693315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11966.834444] Lustre: lustre-MDT0000: new disk, initializing [11966.928869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11966.959587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11972.375735] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [11984.535068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11984.735474] Lustre: 403052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11984.849062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11984.864134] Lustre: Skipped 1 previous similar message [11985.036389] Lustre: lustre-MDT0001: new disk, initializing [11985.220959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11985.295292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11985.316548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11990.489896] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12001.016898] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12008.006808] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12015.385367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12015.688585] Lustre: lustre-OST0000: new disk, initializing [12015.700476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12015.716370] Lustre: 405023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12015.858278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12021.383965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12021.404804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12021.529113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12022.667815] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12033.132537] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12038.846694] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12039.084502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12044.331057] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12044.519879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12047.331211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12047.334092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12047.334377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12047.354465] Lustre: Skipped 1 previous similar message [12047.383470] Lustre: Skipped 1 previous similar message [12052.457682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12052.462521] Lustre: Skipped 1 previous similar message [12052.664631] Lustre: server umount lustre-OST0000 complete [12057.060380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12057.084499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12057.114970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12062.694523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12062.701117] Lustre: Skipped 1 previous similar message [12062.899341] Lustre: server umount lustre-MDT0000 complete [12066.848446] LustreError: 403044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057555 with bad export cookie 14368281792404316156 [12066.859537] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12066.859690] LustreError: 403044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12067.246641] Lustre: server umount lustre-MDT0001 complete [12075.745566] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12077.539900] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:32:44 (1784057564) [12084.768839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.364269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.605766] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12090.852274] LustreError: 407489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12090.871146] LustreError: 407489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12095.972800] LustreError: 407489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.602271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.035902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.864728] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12114.143363] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12121.073745] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12128.353704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12128.887982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12135.520365] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12144.829087] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12146.954797] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12150.908440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12156.897466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12156.909722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.932023] Lustre: Skipped 1 previous similar message [12156.958693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12160.997054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12161.014190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12166.122835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12166.130027] Lustre: Skipped 3 previous similar messages [12171.232122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12171.448802] Lustre: server umount lustre-OST0000 complete [12176.355719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12176.383626] Lustre: Skipped 1 previous similar message [12181.479971] LustreError: 408217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12181.494592] LustreError: 408217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12181.542919] Lustre: server umount lustre-MDT0000 complete [12185.727503] LustreError: 409334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057674 with bad export cookie 14368281792404317038 [12185.744745] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12185.746834] LustreError: 409334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12186.142737] Lustre: server umount lustre-MDT0001 complete [12196.111311] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12198.773561] Key type lgssc unregistered [12199.086944] LNet: 411218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12199.099562] LNetError: 411218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12199.131066] LNet: Removed LNI 192.168.203.151@tcp [12199.984687] Key type .llcrypt unregistered [12199.987900] Key type ._llcrypt unregistered [12225.656441] Key type ._llcrypt registered [12225.659229] Key type .llcrypt registered [12225.862123] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12226.152511] Key type .llcrypt unregistered [12226.154322] Key type ._llcrypt unregistered [12246.902363] Key type ._llcrypt registered [12246.904437] Key type .llcrypt registered [12247.108713] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12248.261723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12248.306420] alg: No test for adler32 (adler32-zlib) [12249.385785] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12249.625198] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12251.321591] Key type lgssc registered [12252.677644] Lustre: Echo OBD driver; http://www.lustre.org/ [12261.121585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12261.142134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12262.655204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12267.457718] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12271.013830] LustreError: 413434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12271.038079] LustreError: 413434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12273.070764] LustreError: 413435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.089398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.523828] LustreError: 413434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.569680] LustreError: 413434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12277.586895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12282.857850] LustreError: 413431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.878270] LustreError: 413431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12283.648935] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12287.971525] LustreError: 413431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.988781] LustreError: 413431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12289.025671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12291.873042] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12297.615119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.080866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12302.721387] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12313.098533] Lustre: server umount lustre-OST0000 complete [12318.690508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12318.691582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12318.703722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12318.753445] Lustre: Skipped 1 previous similar message [12323.818248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.830117] Lustre: Skipped 2 previous similar messages [12327.017699] Lustre: server umount lustre-MDT0000 complete [12328.930586] LustreError: 415451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.957913] LustreError: 415451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12329.952340] Lustre: 412746:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784057802/real 1784057802] req@000000004f75fc1c x1870720126708480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784057818 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12329.974444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12330.862936] LustreError: 413415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057819 with bad export cookie 1768455491136311833 [12330.877751] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12330.887167] LustreError: 413415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12331.331867] Lustre: server umount lustre-MDT0001 complete [12344.544475] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12347.526396] Key type lgssc unregistered [12347.918452] LNet: 416583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12347.936388] LNetError: 416583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12348.969960] LNet: Removed LNI 192.168.203.151@tcp [12350.153104] Key type .llcrypt unregistered [12350.158801] Key type ._llcrypt unregistered [12379.773701] Key type ._llcrypt registered [12379.777867] Key type .llcrypt registered [12379.913466] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12381.162290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12381.180765] alg: No test for adler32 (adler32-zlib) [12382.563169] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12382.887829] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12384.601264] Key type lgssc registered [12385.939893] Lustre: Echo OBD driver; http://www.lustre.org/ [12393.682782] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12393.707173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12409.249418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12414.370730] LustreError: 417974:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.151@tcp: failed processing log, type 4: rc = -110 [12441.056196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12447.573942] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12456.603527] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12463.585769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.129515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12464.192881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12468.516078] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12469.219069] LustreError: 417999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.244257] LustreError: 417999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12474.338688] LustreError: 418000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.369655] LustreError: 417999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.387328] LustreError: 417999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12477.479507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.075713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12482.907893] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12491.150252] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12497.713937] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12504.049883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.073544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12506.150698] Lustre: server umount lustre-OST0000 complete [12511.729119] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12511.750579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12511.770459] Lustre: Skipped 1 previous similar message [12511.776467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12511.782233] Lustre: Skipped 1 previous similar message [12512.514763] Lustre: server umount lustre-MDT0000 complete [12514.288544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12514.293889] LustreError: 419016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.317944] LustreError: 419016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12516.542405] LustreError: 417980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058005 with bad export cookie 15190685807581785798 [12516.543888] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12516.552914] LustreError: 417980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12516.998722] Lustre: server umount lustre-MDT0001 complete [12529.066305] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12531.810932] Key type lgssc unregistered [12532.173448] LNet: 421543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12532.180894] LNetError: 421543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12532.210740] LNet: Removed LNI 192.168.203.151@tcp [12533.230120] Key type .llcrypt unregistered [12533.235111] Key type ._llcrypt unregistered [12557.488487] Key type ._llcrypt registered [12557.491303] Key type .llcrypt registered [12557.613544] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12558.813965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12558.881897] alg: No test for adler32 (adler32-zlib) [12559.933582] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12560.171449] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12561.840777] Key type lgssc registered [12563.038534] Lustre: Echo OBD driver; http://www.lustre.org/ [12570.529837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12570.565653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12572.017821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12576.306644] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12581.348620] LustreError: 422961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.367591] LustreError: 422961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12584.859604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12585.236447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12589.736232] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12597.107921] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12602.911628] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12610.482274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12610.797629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12616.492323] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12616.944486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12623.928182] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12631.521419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12631.528458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12631.547703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12632.062830] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.51@tcp (stopping) [12632.553432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12634.850358] Lustre: server umount lustre-MDT0000 complete [12637.237563] LustreError: 422941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.51@tcp arrived at 1784058126 with bad export cookie 9228006474535754917 [12637.666341] LustreError: 422957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12637.685836] LustreError: 422957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12638.820340] LustreError: 422943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058127 with bad export cookie 9228006474535754021 [12638.832048] LustreError: 422943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12638.836905] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12642.786076] LustreError: 422961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.791086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12642.792201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12642.809136] LustreError: 422961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12642.837582] Lustre: Skipped 1 previous similar message [12642.842728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12642.847282] Lustre: Skipped 3 previous similar messages [12645.034484] Lustre: server umount lustre-OST0000 complete [12648.930526] LustreError: 422961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.948071] LustreError: 422961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12651.337379] Lustre: server umount lustre-MDT0001 complete [12662.531363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12665.507387] Key type lgssc unregistered [12665.826944] LNet: 426645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12665.838421] LNetError: 426645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12665.857318] LNet: Removed LNI 192.168.203.151@tcp [12666.845240] Key type .llcrypt unregistered [12666.848478] Key type ._llcrypt unregistered [12691.078835] Key type ._llcrypt registered [12691.084678] Key type .llcrypt registered [12691.200277] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12692.257740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12692.271150] alg: No test for adler32 (adler32-zlib) [12693.414250] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12693.699350] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12695.408756] Key type lgssc registered [12696.453283] Lustre: Echo OBD driver; http://www.lustre.org/ [12703.257540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12703.277748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12704.655272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12708.054873] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12714.978776] LustreError: 428062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12715.002653] LustreError: 428062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12715.451351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12715.765226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12720.024153] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12727.137805] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12732.316487] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12739.018550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12739.402691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12745.559884] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12749.618738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12751.926939] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12756.962459] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12756.970655] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12756.981437] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12760.039634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12760.049382] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12762.795968] Lustre: server umount lustre-MDT0001 complete [12765.156420] LustreError: 428062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.171539] LustreError: 428062:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12770.275704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12770.279472] LustreError: 428063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.288926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12770.349630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12770.354650] Lustre: Skipped 2 previous similar messages [12775.068908] Lustre: server umount lustre-OST0000 complete [12775.397643] LustreError: 428062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.721626] Lustre: server umount lustre-MDT0000 complete [12790.850261] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [12792.961777] Key type lgssc unregistered [12793.212940] LNet: 431748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12793.230443] LNetError: 431748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12793.245985] LNet: Removed LNI 192.168.203.151@tcp [12793.832273] Key type .llcrypt unregistered [12793.834479] Key type ._llcrypt unregistered [12814.896243] Key type ._llcrypt registered [12814.898418] Key type .llcrypt registered [12815.005705] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [12830.566883] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12831.624835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12831.842212] alg: No test for adler32 (adler32-zlib) [12832.996258] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [12833.288030] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [12835.008155] Key type lgssc registered [12836.159167] Lustre: Echo OBD driver; http://www.lustre.org/ [12875.856545] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [12887.886949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12887.900499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12889.098142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12889.130699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12889.209967] Lustre: lustre-MDT0000: new disk, initializing [12889.291661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12889.310895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12892.824698] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12910.148413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12910.387708] Lustre: lustre-OST0000: new disk, initializing [12910.393815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12910.399222] Lustre: Skipped 1 previous similar message [12910.403295] Lustre: 436782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12910.475222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12916.871363] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12917.885405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12917.897987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12917.929478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12926.841763] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12941.118894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12941.239627] Lustre: 435764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12941.285992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12941.362240] Lustre: lustre-MDT0001: new disk, initializing [12941.442561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12941.499245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12941.514968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12945.282358] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [12954.612296] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12954.839883] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12958.552632] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12958.769738] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12966.375095] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12966.377536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12966.390414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12966.415740] Lustre: Skipped 1 previous similar message [12966.431052] Lustre: Skipped 1 previous similar message [12968.435653] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.51@tcp (stopping) [12969.935713] Lustre: server umount lustre-MDT0001 complete [12976.612704] LustreError: 439229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12976.638093] LustreError: 439229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12981.745072] LustreError: 439229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12981.762212] LustreError: 439229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12986.850022] LustreError: 435774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12986.874568] LustreError: 435774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12989.921495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12989.935359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12989.962369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12991.969210] LustreError: 438119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12991.971675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12993.685388] Lustre: server umount lustre-OST0000 complete [13003.321764] Lustre: server umount lustre-MDT0000 complete [13015.428876] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13017.716623] Key type lgssc unregistered [13017.954834] LNet: 440529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13017.972308] LNetError: 440529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13017.996507] LNet: Removed LNI 192.168.203.151@tcp [13018.782726] Key type .llcrypt unregistered [13018.786813] Key type ._llcrypt unregistered [13059.305408] Key type ._llcrypt registered [13059.308306] Key type .llcrypt registered [13059.422230] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [13059.716302] Key type .llcrypt unregistered [13059.720218] Key type ._llcrypt unregistered [13076.966624] Key type ._llcrypt registered [13076.973673] Key type .llcrypt registered [13077.152885] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [13078.495160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13078.525906] alg: No test for adler32 (adler32-zlib) [13079.634320] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [13079.868717] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [13081.512290] Key type lgssc registered [13082.659693] Lustre: Echo OBD driver; http://www.lustre.org/ [13109.294263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13109.334717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13110.592670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13110.622396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13110.681493] Lustre: lustre-MDT0000: new disk, initializing [13110.775806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13110.809342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13115.602553] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13126.998526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13127.188168] Lustre: 443908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13127.294829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13127.299817] Lustre: Skipped 1 previous similar message [13127.399662] Lustre: lustre-MDT0001: new disk, initializing [13127.517071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13127.567062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13127.585624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13132.824803] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13141.474377] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13147.218873] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13154.208589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13154.531425] Lustre: lustre-OST0000: new disk, initializing [13154.536120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13154.542915] Lustre: 445876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13154.631437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13160.521145] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13164.144886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13164.161751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13164.312373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13169.993579] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13178.853642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13178.859257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.871981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13179.875726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.879545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13179.889446] Lustre: Skipped 1 previous similar message [13179.902773] Lustre: Skipped 1 previous similar message [13183.556329] Lustre: server umount lustre-MDT0000 complete [13187.422150] LustreError: 443898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058676 with bad export cookie 2950552981455396658 [13187.430379] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13187.439798] LustreError: 443898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13187.819284] Lustre: server umount lustre-MDT0001 complete [13202.043560] Lustre: server umount lustre-OST0000 complete [13212.430993] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:51:39 (1784058699) [13220.154857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13220.649350] LustreError: 447905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.668096] LustreError: 447905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13220.710826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13225.186648] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13225.962112] LustreError: 447906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13231.074268] LustreError: 447905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13234.069665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13234.458578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13238.706245] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13245.912939] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13252.041697] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13271.993067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13272.242246] Lustre: Found index 0 for lustre-OST0000, updating log [13272.459956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13278.000783] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13285.952090] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13291.950891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13297.130934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13297.137543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13297.143051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13297.150049] Lustre: Skipped 1 previous similar message [13298.540686] Lustre: server umount lustre-OST0000 complete [13314.544519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13314.552352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13314.556298] Lustre: Skipped 1 previous similar message [13314.570992] Lustre: Skipped 4 previous similar messages [13319.653218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13319.661058] Lustre: Skipped 1 previous similar message [13320.985817] Lustre: server umount lustre-MDT0000 complete [13324.774709] LustreError: 448634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.793397] LustreError: 448634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13325.309490] LustreError: 449748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784058814 with bad export cookie 2950552981455398051 [13325.318733] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13325.326090] LustreError: 449748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13325.736861] Lustre: server umount lustre-MDT0001 complete [13329.767399] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [13335.073264] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:53:42 (1784058822) [13951.257451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.706995] LustreError: 454325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13951.782637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13956.318197] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13957.089561] LustreError: 454326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13961.189296] LustreError: 454325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13964.427190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13964.807942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13968.844695] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [13975.894473] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13981.395440] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14001.204544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14001.356748] Lustre: Found index 0 for lustre-OST0000, updating log [14001.577451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14006.905236] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14008.672237] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [14014.015908] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14016.102847] Lustre: server umount lustre-OST0000 complete [14021.098513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14021.098852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14021.101451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14021.108462] Lustre: Skipped 1 previous similar message [14022.114828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14022.134461] Lustre: Skipped 2 previous similar messages [14024.161961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14024.184644] Lustre: Skipped 3 previous similar messages [14028.264862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14028.268843] Lustre: Skipped 6 previous similar messages [14034.400171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14034.610641] Lustre: server umount lustre-MDT0000 complete [14035.436487] LustreError: 456793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14035.461693] LustreError: 456793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14038.203857] LustreError: 455059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784059527 with bad export cookie 2950552981455399136 [14038.210833] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14038.215369] LustreError: 455059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14038.493288] Lustre: server umount lustre-MDT0001 complete [14066.781119] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [14072.160366] LNet: 1 local NIs in recovery (showing 1): 192.168.203.151@tcp [14078.929279] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14117.067655] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:06:44 (1784059604) [14133.211685] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14135.750471] Key type lgssc unregistered [14136.001475] LNet: 460941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14136.008257] LNetError: 460941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14264.038712] LNet: Removed LNI 192.168.203.151@tcp [14265.022125] Key type .llcrypt unregistered [14265.026111] Key type ._llcrypt unregistered [14282.967261] Key type ._llcrypt registered [14282.968489] Key type .llcrypt registered [14283.044366] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14284.326980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14284.531141] alg: No test for adler32 (adler32-zlib) [14285.665652] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [14285.937509] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [14287.649464] Key type lgssc registered [14288.870937] Lustre: Echo OBD driver; http://www.lustre.org/ [14293.713328] LNet: Added LNI 192.168.203.151@tcp100 [8/256/0/180] [14294.568729] LNet: Added LNI 192.168.203.151@tcp101 [8/256/0/180] [14295.724118] LNet: Added LNI 192.168.203.151@tcp102 [8/256/0/180] [14296.613459] LNet: Added LNI 192.168.203.151@tcp103 [8/256/0/180] [14297.416348] LNet: Added LNI 192.168.203.151@tcp104 [8/256/0/180] [14298.379664] LNet: Added LNI 192.168.203.151@tcp105 [8/256/0/180] [14299.153535] LNet: Added LNI 192.168.203.151@tcp106 [8/256/0/180] [14300.031346] LNet: Added LNI 192.168.203.151@tcp107 [8/256/0/180] [14300.986523] LNet: Added LNI 192.168.203.151@tcp108 [8/256/0/180] [14301.773896] LNet: Added LNI 192.168.203.151@tcp109 [8/256/0/180] [14302.574467] LNet: Added LNI 192.168.203.151@tcp110 [8/256/0/180] [14303.455976] LNet: Added LNI 192.168.203.151@tcp111 [8/256/0/180] [14304.223565] LNet: Added LNI 192.168.203.151@tcp112 [8/256/0/180] [14305.006789] LNet: Added LNI 192.168.203.151@tcp113 [8/256/0/180] [14305.855461] LNet: Added LNI 192.168.203.151@tcp114 [8/256/0/180] [14306.737376] LNet: Added LNI 192.168.203.151@tcp115 [8/256/0/180] [14307.629631] LNet: Added LNI 192.168.203.151@tcp116 [8/256/0/180] [14308.503735] LNet: Added LNI 192.168.203.151@tcp117 [8/256/0/180] [14309.471578] LNet: Added LNI 192.168.203.151@tcp118 [8/256/0/180] [14310.255527] LNet: Added LNI 192.168.203.151@tcp119 [8/256/0/180] [14311.123575] LNet: Added LNI 192.168.203.151@tcp120 [8/256/0/180] [14311.987810] LNet: Added LNI 192.168.203.151@tcp121 [8/256/0/180] [14312.790492] LNet: Added LNI 192.168.203.151@tcp122 [8/256/0/180] [14313.760810] LNet: Added LNI 192.168.203.151@tcp123 [8/256/0/180] [14314.681513] LNet: Added LNI 192.168.203.151@tcp124 [8/256/0/180] [14315.415593] LNet: Added LNI 192.168.203.151@tcp125 [8/256/0/180] [14316.267554] LNet: Added LNI 192.168.203.151@tcp126 [8/256/0/180] [14317.145557] LNet: Added LNI 192.168.203.151@tcp127 [8/256/0/180] [14318.055287] LNet: Added LNI 192.168.203.151@tcp128 [8/256/0/180] [14318.880663] LNet: Added LNI 192.168.203.151@tcp129 [8/256/0/180] [14319.692300] LNet: Added LNI 192.168.203.151@tcp130 [8/256/0/180] [14320.582958] LNet: Added LNI 192.168.203.151@tcp131 [8/256/0/180] [14321.413567] LNet: Added LNI 192.168.203.151@tcp132 [8/256/0/180] [14322.262198] LNet: Added LNI 192.168.203.151@tcp133 [8/256/0/180] [14323.127901] LNet: Added LNI 192.168.203.151@tcp134 [8/256/0/180] [14323.971471] LNet: Added LNI 192.168.203.151@tcp135 [8/256/0/180] [14331.080373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14331.102730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14332.233543] Lustre: 463975:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14332.316040] LNetError: 463997:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14332.323306] LNetError: Protocol error connecting to 192.168.203.151@tcp117 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14332.335820] LNetError: Skipped 4 previous similar messages [14332.421254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14332.449397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14332.511167] Lustre: lustre-MDT0000: new disk, initializing [14332.580390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14332.594248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14333.092847] LNet: 6 local NIs in recovery (showing 6): 192.168.203.151@tcp121, 192.168.203.151@tcp125, 192.168.203.151@tcp124, 192.168.203.151@tcp126, 192.168.203.151@tcp129, 192.168.203.151@tcp128 [14333.096745] LNetError: 461434:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14333.140419] LNetError: 461434:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14333.151536] LNetError: Protocol error connecting to 192.168.203.151@tcp100 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14333.163470] LNetError: Skipped 6 previous similar messages [14336.944657] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14346.905064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14347.049130] Lustre: 463992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14347.135739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14347.141261] Lustre: Skipped 1 previous similar message [14347.241400] Lustre: lustre-MDT0001: new disk, initializing [14347.286144] Lustre: 464795:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14347.291157] Lustre: 464795:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14347.399842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14347.454700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14347.479157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14352.098445] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14356.017915] Lustre: 465221:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14356.024241] Lustre: 465221:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14361.284709] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14362.730796] LNetError: 463999:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14362.744286] LNetError: 463999:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [14362.752511] LNetError: Protocol error connecting to 192.168.203.151@tcp123 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14364.780595] LNetError: 461435:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14364.801881] LNetError: 461435:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14364.818425] LNetError: Protocol error connecting to 192.168.203.151@tcp100 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14364.838319] LNetError: Skipped 1 previous similar message [14366.585326] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14368.865353] LNetError: 464006:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14368.874986] LNetError: 464006:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14368.884422] LNetError: Protocol error connecting to 192.168.203.151@tcp119 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14368.895573] LNetError: Skipped 1 previous similar message [14372.967898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14373.330478] Lustre: lustre-OST0000: new disk, initializing [14373.334469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14373.342954] Lustre: 465968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14373.419634] Lustre: 465973:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14373.428077] Lustre: 465973:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14373.488165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14377.059827] LNetError: 461435:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14377.067684] LNetError: 461435:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14377.074097] LNetError: Protocol error connecting to 192.168.203.151@tcp100 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14377.083790] LNetError: Skipped 7 previous similar messages [14379.425256] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14383.652811] Lustre: 466492:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.151@tcp: can't add NID 192.168.203.151@tcp130: rc = -75 [14383.662755] Lustre: 466492:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14383.697857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14383.715071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14383.831233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14387.562078] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14393.449354] LNetError: 463997:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14393.461962] LNetError: 463997:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14393.468762] LNetError: Protocol error connecting to 192.168.203.151@tcp119 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14393.481753] LNetError: Skipped 11 previous similar messages [14399.459859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14399.466904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14399.472727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14399.492791] Lustre: Skipped 1 previous similar message [14399.506785] Lustre: Skipped 1 previous similar message [14400.618031] Lustre: server umount lustre-OST0000 complete [14408.677486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14408.690525] LustreError: Skipped 1 previous similar message [14408.704668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14408.728279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14411.470560] Lustre: server umount lustre-MDT0000 complete [14416.327842] LustreError: 464794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784059905 with bad export cookie 7446721858707110514 [14416.334470] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14416.352287] LustreError: 464794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14416.768773] Lustre: server umount lustre-MDT0001 complete [14426.209434] LNetError: 461432:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.151 [14426.220376] LNetError: 461432:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [14426.229987] LNetError: Protocol error connecting to 192.168.203.151@tcp123 at host 192.168.203.151:988: is it running a compatible version of Lustre? [14426.240933] LNetError: Skipped 15 previous similar messages [14429.811614] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14432.663906] Key type lgssc unregistered [14433.052456] LNet: 467880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14433.057020] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14433.076743] LNet: Removed LNI 192.168.203.151@tcp [14434.144165] LNet: Removed LNI 192.168.203.151@tcp100 [14434.155071] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14434.164774] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14434.172266] LNet: Removed LNI 192.168.203.151@tcp101 [14435.232132] LNet: Removed LNI 192.168.203.151@tcp102 [14435.239202] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14435.254852] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14435.260908] LNet: Removed LNI 192.168.203.151@tcp103 [14436.320398] LNet: Removed LNI 192.168.203.151@tcp104 [14437.344168] LNet: Removed LNI 192.168.203.151@tcp105 [14437.346252] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14437.353911] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14437.362813] LNet: Removed LNI 192.168.203.151@tcp106 [14438.432175] LNet: Removed LNI 192.168.203.151@tcp107 [14439.456447] LNet: Removed LNI 192.168.203.151@tcp108 [14440.482438] LNet: Removed LNI 192.168.203.151@tcp109 [14441.505342] LNet: Removed LNI 192.168.203.151@tcp110 [14441.510830] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14441.519222] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14441.531121] LNet: Removed LNI 192.168.203.151@tcp111 [14442.592167] LNet: Removed LNI 192.168.203.151@tcp112 [14443.618448] LNet: Removed LNI 192.168.203.151@tcp113 [14444.641526] LNet: Removed LNI 192.168.203.151@tcp114 [14445.664147] LNet: Removed LNI 192.168.203.151@tcp115 [14446.688147] LNet: Removed LNI 192.168.203.151@tcp116 [14447.712175] LNet: Removed LNI 192.168.203.151@tcp117 [14448.736230] LNet: Removed LNI 192.168.203.151@tcp118 [14449.760184] LNet: Removed LNI 192.168.203.151@tcp119 [14449.767580] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14449.779676] LNetError: 467880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14449.788656] LNet: Removed LNI 192.168.203.151@tcp120 [14450.848150] LNet: Removed LNI 192.168.203.151@tcp121 [14451.872159] LNet: Removed LNI 192.168.203.151@tcp122 [14452.896155] LNet: Removed LNI 192.168.203.151@tcp123 [14453.920174] LNet: Removed LNI 192.168.203.151@tcp124 [14454.944474] LNet: Removed LNI 192.168.203.151@tcp125 [14455.971279] LNet: Removed LNI 192.168.203.151@tcp126 [14456.992418] LNet: Removed LNI 192.168.203.151@tcp127 [14458.016322] LNet: Removed LNI 192.168.203.151@tcp128 [14459.040127] LNet: Removed LNI 192.168.203.151@tcp129 [14459.043332] LNet: Removed LNI 192.168.203.151@tcp130 [14459.050551] LNet: Removed LNI 192.168.203.151@tcp131 [14459.054572] LNet: Removed LNI 192.168.203.151@tcp132 [14459.057500] LNet: Removed LNI 192.168.203.151@tcp133 [14459.067162] LNet: Removed LNI 192.168.203.151@tcp134 [14459.087180] LNet: Removed LNI 192.168.203.151@tcp135 [14459.982926] Key type .llcrypt unregistered [14459.984747] Key type ._llcrypt unregistered [14483.188409] Key type ._llcrypt registered [14483.191539] Key type .llcrypt registered [14483.287224] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [14499.483706] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14500.947949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14501.014464] alg: No test for adler32 (adler32-zlib) [14502.206114] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [14502.505570] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [14504.192175] Key type lgssc registered [14505.415453] Lustre: Echo OBD driver; http://www.lustre.org/ [14542.373672] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:13:49 (1784060029) [14561.543687] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [14563.926198] Key type lgssc unregistered [14564.184712] LNet: 471970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14564.193481] LNetError: 471970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14564.205959] LNet: Removed LNI 192.168.203.151@tcp [14564.942814] Key type .llcrypt unregistered [14564.949649] Key type ._llcrypt unregistered [14582.660978] Key type ._llcrypt registered [14582.667511] Key type .llcrypt registered [14582.779974] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14583.983358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14584.027160] alg: No test for adler32 (adler32-zlib) [14585.127895] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [14585.452737] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [14587.136455] Key type lgssc registered [14588.169582] Lustre: Echo OBD driver; http://www.lustre.org/ [14597.301381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14597.346931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14598.637871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14598.692649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14598.775894] Lustre: lustre-MDT0000: new disk, initializing [14598.888311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14598.911648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14603.441516] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14613.812517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14613.953546] Lustre: 473182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14614.044236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14614.049783] Lustre: Skipped 1 previous similar message [14614.105171] Lustre: lustre-MDT0001: new disk, initializing [14614.193556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14614.228770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14614.245976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14618.093325] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14627.762495] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14634.098954] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14641.083360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14641.334370] Lustre: lustre-OST0000: new disk, initializing [14641.338397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14641.343587] Lustre: 475153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14641.437382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14647.628866] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [14651.494232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14651.509371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14651.617241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14657.729709] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14662.238458] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing lnet_if_list [14667.486900] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14667.526660] LNet: Added LNI 192.168.203.151@tcp10 [8/256/0/180] [14672.628971] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14672.655098] LNet: Added LNI 192.168.203.151@tcp11 [8/256/0/180] [14677.657198] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14677.713442] LNet: Added LNI 192.168.203.151@tcp12 [8/256/0/180] [14682.722890] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14682.755551] LNet: Added LNI 192.168.203.151@tcp13 [8/256/0/180] [14687.328433] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14687.357980] LNet: Added LNI 192.168.203.151@tcp14 [8/256/0/180] [14691.671723] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14691.692780] LNet: Added LNI 192.168.203.151@tcp15 [8/256/0/180] [14696.448908] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14696.482770] LNet: Added LNI 192.168.203.151@tcp16 [8/256/0/180] [14701.446427] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14701.483501] LNet: Added LNI 192.168.203.151@tcp17 [8/256/0/180] [14706.212404] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14706.238780] LNet: Added LNI 192.168.203.151@tcp18 [8/256/0/180] [14710.910280] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14710.934000] LNet: Added LNI 192.168.203.151@tcp19 [8/256/0/180] [14715.718170] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14715.752263] LNet: Added LNI 192.168.203.151@tcp20 [8/256/0/180] [14720.608588] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14720.629822] LNet: Added LNI 192.168.203.151@tcp21 [8/256/0/180] [14725.820393] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14725.850196] LNet: Added LNI 192.168.203.151@tcp22 [8/256/0/180] [14731.396869] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14731.421656] LNet: Added LNI 192.168.203.151@tcp23 [8/256/0/180] [14737.506735] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14737.557429] LNet: Added LNI 192.168.203.151@tcp24 [8/256/0/180] [14744.145878] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14744.196856] LNet: Added LNI 192.168.203.151@tcp25 [8/256/0/180] [14752.691727] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14752.750644] LNet: Added LNI 192.168.203.151@tcp26 [8/256/0/180] [14758.943328] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14758.962642] LNet: Added LNI 192.168.203.151@tcp27 [8/256/0/180] [14765.546256] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14765.593163] LNet: Added LNI 192.168.203.151@tcp28 [8/256/0/180] [14771.882393] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14771.935085] LNet: Added LNI 192.168.203.151@tcp29 [8/256/0/180] [14778.147925] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14778.174712] LNet: Added LNI 192.168.203.151@tcp30 [8/256/0/180] [14783.699981] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14783.735614] LNet: Added LNI 192.168.203.151@tcp31 [8/256/0/180] [14789.676805] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14789.702630] LNet: Added LNI 192.168.203.151@tcp32 [8/256/0/180] [14796.560244] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14796.594437] LNet: Added LNI 192.168.203.151@tcp33 [8/256/0/180] [14803.584733] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14803.627950] LNet: Added LNI 192.168.203.151@tcp34 [8/256/0/180] [14810.518087] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14810.558732] LNet: Added LNI 192.168.203.151@tcp35 [8/256/0/180] [14818.148115] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14818.198313] LNet: Added LNI 192.168.203.151@tcp36 [8/256/0/180] [14825.908391] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14825.940516] LNet: Added LNI 192.168.203.151@tcp37 [8/256/0/180] [14832.597954] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14832.640114] LNet: Added LNI 192.168.203.151@tcp38 [8/256/0/180] [14838.595564] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14838.634036] LNet: Added LNI 192.168.203.151@tcp39 [8/256/0/180] [14846.313834] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14846.345804] LNet: Added LNI 192.168.203.151@tcp40 [8/256/0/180] [14854.916988] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14854.960281] LNet: Added LNI 192.168.203.151@tcp41 [8/256/0/180] [14863.513689] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14863.548698] LNet: Added LNI 192.168.203.151@tcp42 [8/256/0/180] [14872.428440] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14872.484108] LNet: Added LNI 192.168.203.151@tcp43 [8/256/0/180] [14882.051500] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14882.073593] LNet: Added LNI 192.168.203.151@tcp44 [8/256/0/180] [14891.069617] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14891.101653] LNet: Added LNI 192.168.203.151@tcp45 [8/256/0/180] [14899.370848] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14899.397523] LNet: Added LNI 192.168.203.151@tcp46 [8/256/0/180] [14908.434230] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14908.466916] LNet: Added LNI 192.168.203.151@tcp47 [8/256/0/180] [14917.007945] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14917.084751] LNet: Added LNI 192.168.203.151@tcp48 [8/256/0/180] [14926.455942] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14926.485795] LNet: Added LNI 192.168.203.151@tcp49 [8/256/0/180] [14936.027606] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14936.069191] LNet: Added LNI 192.168.203.151@tcp50 [8/256/0/180] [14957.538335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14957.555715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14957.572861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14959.076264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14959.079087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14962.204939] Lustre: server umount lustre-MDT0000 complete [14966.284609] LustreError: 473172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060455 with bad export cookie 17403730388873825113 [14966.290494] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14966.313680] LustreError: 473172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14967.085494] Lustre: server umount lustre-MDT0001 complete [14985.440086] Lustre: 472453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784060458/real 1784060458] req@00000000fbbcaffe x1870722576324992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14985.458996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14985.471022] Lustre: Skipped 1 previous similar message [14987.415084] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [14987.755519] Lustre: 472454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784060460/real 1784060460] req@00000000b59dd29e x1870722576325120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14990.817121] Lustre: 472455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784060463/real 1784060463] req@00000000168cd9ea x1870722576325248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14992.864185] Lustre: 472452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784060465/real 1784060465] req@000000006685d4f0 x1870722576325504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14996.105786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14998.048142] Lustre: 472452:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784060470/real 1784060470] req@00000000166b6c68 x1870722576325632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784060486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15012.464521] LustreError: 475147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15012.495158] LustreError: 475147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15012.521868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15014.579406] LustreError: 475148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15014.591519] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15016.608501] LustreError: 475147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15017.002674] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15022.052574] LustreError: 475148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15027.174633] LustreError: 475147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15027.223260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15027.626318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15030.711226] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15032.778303] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15040.646621] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15047.099653] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15066.608706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15066.614258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15066.634137] Lustre: Skipped 1 previous similar message [15066.656864] Lustre: Skipped 3 previous similar messages [15071.718677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15071.727098] Lustre: Skipped 1 previous similar message [15076.841087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15076.852539] Lustre: Skipped 1 previous similar message [15079.392212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15079.599482] Lustre: server umount lustre-OST0000 complete [15084.007087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15084.028705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15087.082713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15087.102705] Lustre: Skipped 2 previous similar messages [15090.039094] Lustre: server umount lustre-MDT0000 complete [15092.196429] LustreError: 499394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15092.213964] LustreError: 499394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15093.711333] LustreError: 473174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060582 with bad export cookie 17403730388873827738 [15093.713605] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15093.729434] LustreError: 473174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15094.058306] Lustre: server umount lustre-MDT0001 complete [15104.260843] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15106.646759] Key type lgssc unregistered [15106.916691] LNet: 502173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15106.923135] LNetError: 502173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15107.936606] LNet: Removed LNI 192.168.203.151@tcp [15107.940897] LNetError: 502173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15107.952029] LNet: Removed LNI 192.168.203.151@tcp10 [15107.956911] LNet: Removed LNI 192.168.203.151@tcp11 [15107.962979] LNet: Removed LNI 192.168.203.151@tcp12 [15107.970393] LNet: Removed LNI 192.168.203.151@tcp13 [15107.979043] LNet: Removed LNI 192.168.203.151@tcp14 [15107.989196] LNet: Removed LNI 192.168.203.151@tcp15 [15107.999216] LNet: Removed LNI 192.168.203.151@tcp16 [15108.006977] LNet: Removed LNI 192.168.203.151@tcp17 [15108.010708] LNet: Removed LNI 192.168.203.151@tcp18 [15108.025938] LNet: Removed LNI 192.168.203.151@tcp19 [15108.028930] LNet: Removed LNI 192.168.203.151@tcp20 [15108.037894] LNet: Removed LNI 192.168.203.151@tcp21 [15108.046181] LNet: Removed LNI 192.168.203.151@tcp22 [15108.047965] LNet: Removed LNI 192.168.203.151@tcp23 [15108.059229] LNet: Removed LNI 192.168.203.151@tcp24 [15108.061629] LNet: Removed LNI 192.168.203.151@tcp25 [15108.065805] LNet: Removed LNI 192.168.203.151@tcp26 [15108.070959] LNet: Removed LNI 192.168.203.151@tcp27 [15108.075492] LNet: Removed LNI 192.168.203.151@tcp28 [15108.079831] LNet: Removed LNI 192.168.203.151@tcp29 [15108.082979] LNet: Removed LNI 192.168.203.151@tcp30 [15108.096377] LNet: Removed LNI 192.168.203.151@tcp31 [15108.100406] LNet: Removed LNI 192.168.203.151@tcp32 [15108.108937] LNet: Removed LNI 192.168.203.151@tcp33 [15108.117313] LNet: Removed LNI 192.168.203.151@tcp34 [15108.121582] LNet: Removed LNI 192.168.203.151@tcp35 [15108.127549] LNet: Removed LNI 192.168.203.151@tcp36 [15108.135879] LNet: Removed LNI 192.168.203.151@tcp37 [15108.141966] LNet: Removed LNI 192.168.203.151@tcp38 [15108.147153] LNet: Removed LNI 192.168.203.151@tcp39 [15108.150371] LNet: Removed LNI 192.168.203.151@tcp40 [15108.153488] LNet: Removed LNI 192.168.203.151@tcp41 [15108.159171] LNet: Removed LNI 192.168.203.151@tcp42 [15108.166475] LNet: Removed LNI 192.168.203.151@tcp43 [15108.171259] LNet: Removed LNI 192.168.203.151@tcp44 [15108.179464] LNet: Removed LNI 192.168.203.151@tcp45 [15108.184196] LNet: Removed LNI 192.168.203.151@tcp46 [15108.189680] LNet: Removed LNI 192.168.203.151@tcp47 [15108.197402] LNet: Removed LNI 192.168.203.151@tcp48 [15108.202606] LNet: Removed LNI 192.168.203.151@tcp49 [15108.245066] LNet: Removed LNI 192.168.203.151@tcp50 [15109.172429] Key type .llcrypt unregistered [15109.175688] Key type ._llcrypt unregistered [15131.353354] Key type ._llcrypt registered [15131.355418] Key type .llcrypt registered [15131.488982] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [15147.916859] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15149.567664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15149.619133] alg: No test for adler32 (adler32-zlib) [15150.785637] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [15151.033437] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [15152.705185] Key type lgssc registered [15153.842433] Lustre: Echo OBD driver; http://www.lustre.org/ [15187.720829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15187.760069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15189.007592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15189.077813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15189.214600] Lustre: lustre-MDT0000: new disk, initializing [15189.322707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15189.339656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15193.967787] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15204.541235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15204.693488] Lustre: 505369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15204.794550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15204.808417] Lustre: Skipped 1 previous similar message [15204.932696] Lustre: lustre-MDT0001: new disk, initializing [15205.072425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15205.115374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15205.135691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15210.059784] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15218.668696] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15223.868305] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15230.759076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15230.957358] Lustre: *** cfs_fail_loc=90f, val=0*** [15231.049226] Lustre: lustre-OST0000: new disk, initializing [15231.054242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15231.066779] Lustre: 507385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15231.148693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15236.623524] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15237.657154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15237.674330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15237.774819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15244.266226] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15252.961530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15252.966543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15252.974406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15253.476466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15253.478136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15253.485127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15256.569386] Lustre: server umount lustre-OST0000 complete [15261.666450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15261.671443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15261.694066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15261.704308] Lustre: Skipped 1 previous similar message [15263.720776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15263.722280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15263.739739] Lustre: Skipped 1 previous similar message [15265.812712] Lustre: server umount lustre-MDT0000 complete [15269.194133] LustreError: 506164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060758 with bad export cookie 756453202809608483 [15269.200379] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15269.202369] LustreError: 506164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15269.514325] Lustre: server umount lustre-MDT0001 complete [15278.746817] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15281.329206] Key type lgssc unregistered [15281.627711] LNet: 509291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15281.633158] LNetError: 509291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15281.646246] LNet: Removed LNI 192.168.203.151@tcp [15282.472514] Key type .llcrypt unregistered [15282.475580] Key type ._llcrypt unregistered [15299.439214] Key type ._llcrypt registered [15299.445269] Key type .llcrypt registered [15299.561542] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15300.551853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15300.571612] alg: No test for adler32 (adler32-zlib) [15301.600703] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [15301.816949] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [15303.536157] Key type lgssc registered [15304.630976] Lustre: Echo OBD driver; http://www.lustre.org/ [15311.459305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15311.473696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15312.906626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15316.728836] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15323.106957] LustreError: 510542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15323.118632] LustreError: 510542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15324.015598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15324.329716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15328.390319] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15335.138952] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15340.421563] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15347.695031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15348.032111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15353.518445] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15360.301931] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15369.459991] loop0: detected capacity change from 0 to 204800000 [15369.697511] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15369.719767] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15369.745407] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15375.334712] loop0: detected capacity change from 0 to 204800000 [15375.465428] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15375.474102] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15375.486505] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15380.801405] loop0: detected capacity change from 0 to 204800000 [15385.850249] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15386.011065] Lustre: fs15246-OST0000: new disk, initializing [15386.017332] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15386.025179] Lustre: 514135:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15386.075118] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15391.444426] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15400.436337] loop1: detected capacity change from 0 to 204800000 [15405.716754] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15405.850148] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15405.914692] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15406.000267] Lustre: fs15246-MDT0000: new disk, initializing [15406.167115] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15406.208117] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15406.321777] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15407.395061] Lustre: 515255:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000082200f49 x1870723326980352/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784060912 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15410.049609] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15416.490095] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15416.502299] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15416.528278] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15433.195353] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15433.222325] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15435.935095] Lustre: server umount fs15246-MDT0000 complete [15438.317733] LustreError: 510538:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15443.426632] LustreError: 510537:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15445.839448] Lustre: server umount fs15246-OST0000 complete [15457.766040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15457.775065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15457.793429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15458.794419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15458.801521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15463.244114] Lustre: server umount lustre-OST0000 complete [15467.489681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15467.499135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15467.525167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15467.527568] Lustre: Skipped 1 previous similar message [15472.924906] Lustre: server umount lustre-MDT0000 complete [15474.145876] LustreError: 512384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15474.170919] LustreError: 512384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15476.644772] LustreError: 512385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784060965 with bad export cookie 12358939867218169989 [15476.651974] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15476.681318] LustreError: 512385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15477.061837] Lustre: server umount lustre-MDT0001 complete [15487.418416] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15489.897193] Key type lgssc unregistered [15490.155966] LNet: 518351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15490.171108] LNetError: 518351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15490.205868] LNet: Removed LNI 192.168.203.151@tcp [15490.961223] Key type .llcrypt unregistered [15490.963127] Key type ._llcrypt unregistered [15510.180121] Key type ._llcrypt registered [15510.182695] Key type .llcrypt registered [15510.303251] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15511.216765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15511.297313] alg: No test for adler32 (adler32-zlib) [15512.459965] Lustre: Lustre: Build Version: 2.17.54_162_g336b9c0 [15512.704981] LNet: Added LNI 192.168.203.151@tcp [8/256/0/180] [15514.392222] Key type lgssc registered [15515.499232] Lustre: Echo OBD driver; http://www.lustre.org/ [15550.078498] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [15560.246784] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15595.903284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15595.930442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15597.161504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15597.248360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15597.360350] Lustre: lustre-MDT0000: new disk, initializing [15597.421630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15597.452299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15601.764822] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15611.838196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15611.974446] Lustre: 522814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15612.016581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15612.020220] Lustre: Skipped 1 previous similar message [15612.089963] Lustre: lustre-MDT0001: new disk, initializing [15612.210909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15612.262964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15612.307559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15616.850177] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15626.502080] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15632.431058] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15640.303059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15640.749671] Lustre: lustre-OST0000: new disk, initializing [15640.760433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15640.767153] Lustre: 524785:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15640.922121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15648.085769] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_default_debug -1 all [15648.345871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15648.363442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15648.557949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15658.668604] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15663.154132] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15663.512957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15669.056940] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15669.322177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15674.345938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15674.349245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15674.376498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15674.388532] Lustre: Skipped 1 previous similar message [15677.498958] Lustre: server umount lustre-OST0000 complete [15684.065060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15684.070065] LustreError: Skipped 1 previous similar message [15684.072954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15684.079863] Lustre: Skipped 1 previous similar message [15684.085986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15689.700913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15689.712049] Lustre: Skipped 2 previous similar messages [15694.819136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15694.825708] Lustre: Skipped 1 previous similar message [15695.328182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15695.443142] Lustre: server umount lustre-MDT0000 complete [15698.604861] LustreError: 522807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784061187 with bad export cookie 275923714312606758 [15698.609992] LustreError: MGC192.168.203.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15698.613040] LustreError: 522807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15698.983205] Lustre: server umount lustre-MDT0001 complete [15722.319492] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing set_hostid [15732.806229] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing load_modules_local [15763.574081] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15464 sec ========= 16:34:10 (1784061250) [15765.139744] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:34:12 (1784061252) === [15768.559245] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:34:15 (1784061255) === [15807.033798] Lustre: DEBUG MARKER: oleg351-server.virtnet: executing unload_modules_local [15810.176963] Key type lgssc unregistered [15810.467585] LNet: 531431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15810.474617] LNetError: 531431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15810.500154] LNet: Removed LNI 192.168.203.151@tcp [15811.320922] Key type .llcrypt unregistered [15811.324247] Key type ._llcrypt unregistered