[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 523679168 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002221] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006865] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007036] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009137] LSM: Security Framework initializing [ 0.010063] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.019725] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023784] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027043] *** VALIDATE tmpfs *** [ 0.028412] *** VALIDATE proc *** [ 0.029225] *** VALIDATE cgroup *** [ 0.030010] *** VALIDATE cgroup2 *** [ 0.032264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036009] Speculative Store Bypass: Vulnerable [ 0.038956] debug: unmapping init [mem 0xffffffffa4a59000-0xffffffffa4a60fff] [ 0.040835] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042139] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043027] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047053] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049011] ... event mask: 000000070000000f [ 0.050309] rcu: Hierarchical SRCU implementation. [ 0.052220] smp: Bringing up secondary CPUs ... [ 0.053644] x86: Booting SMP configuration: [ 0.054049] .... node #0, CPUs: #1 #2 #3 [ 0.057811] smp: Brought up 1 node, 4 CPUs [ 0.059097] smpboot: Max logical packages: 1 [ 0.060019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.198680] node 0 deferred pages initialised in 136ms [ 0.202015] devtmpfs: initialized [ 0.203232] x86/mm: Memory block size: 128MB [ 0.206026] gcov: version magic: 0x41383552 [ 0.208355] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.210260] pinctrl core: initialized pinctrl subsystem [ 0.211236] [ 0.211814] ************************************************************* [ 0.212017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.213024] ** ** [ 0.214019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.215016] ** ** [ 0.216014] ** This means that this kernel is built to expose internal ** [ 0.217013] ** IOMMU data structures, which may compromise security on ** [ 0.218020] ** your system. ** [ 0.219017] ** ** [ 0.220014] ** If you see this message and you are not debugging the ** [ 0.221014] ** kernel, report this immediately to your vendor! ** [ 0.222015] ** ** [ 0.223016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224014] ************************************************************* [ 0.225764] NET: Registered protocol family 16 [ 0.226542] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.227081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.228069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.229419] cpuidle: using governor menu [ 0.232754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.235509] PCI: Using configuration type 1 for base access [ 0.237122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.246148] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.249049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.252195] cryptd: max_cpu_qlen set to 1000 [ 0.255303] ACPI: Added _OSI(Module Device) [ 0.257017] ACPI: Added _OSI(Processor Device) [ 0.259018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.265162] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.272452] ACPI: Interpreter enabled [ 0.273079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.275036] ACPI: Using IOAPIC for interrupt routing [ 0.277138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.280645] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.292280] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.295060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.297021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.302105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308394] acpiphp: Slot [2] registered [ 0.310138] acpiphp: Slot [5] registered [ 0.311168] acpiphp: Slot [6] registered [ 0.313152] acpiphp: Slot [7] registered [ 0.314198] acpiphp: Slot [8] registered [ 0.316150] acpiphp: Slot [9] registered [ 0.318373] acpiphp: Slot [10] registered [ 0.319115] acpiphp: Slot [3] registered [ 0.321129] acpiphp: Slot [4] registered [ 0.322093] acpiphp: Slot [11] registered [ 0.324124] acpiphp: Slot [12] registered [ 0.326149] acpiphp: Slot [13] registered [ 0.327128] acpiphp: Slot [14] registered [ 0.329131] acpiphp: Slot [15] registered [ 0.330117] acpiphp: Slot [16] registered [ 0.332102] acpiphp: Slot [17] registered [ 0.333115] acpiphp: Slot [18] registered [ 0.335161] acpiphp: Slot [19] registered [ 0.337139] acpiphp: Slot [20] registered [ 0.339112] acpiphp: Slot [21] registered [ 0.340100] acpiphp: Slot [22] registered [ 0.342135] acpiphp: Slot [23] registered [ 0.343107] acpiphp: Slot [24] registered [ 0.345115] acpiphp: Slot [25] registered [ 0.347119] acpiphp: Slot [26] registered [ 0.348090] acpiphp: Slot [27] registered [ 0.349102] acpiphp: Slot [28] registered [ 0.351099] acpiphp: Slot [29] registered [ 0.353159] acpiphp: Slot [30] registered [ 0.354180] acpiphp: Slot [31] registered [ 0.356080] PCI host bridge to bus 0000:00 [ 0.357029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.360032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.362031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.366026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.369027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.371027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.373160] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.376000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379598] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389838] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.395750] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.401017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.404021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.409785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.413022] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416311] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.422019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.438015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.444878] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.448906] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.457016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.465018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.490865] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.509021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.550272] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.564022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.573015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.595019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.608513] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.619016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.629017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.670000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.678018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.688022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.716021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.728557] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.734000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.744016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.764017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.774916] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.777897] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.780360] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.782445] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.784224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790113] iommu: Default domain type: Passthrough [ 0.791461] SCSI subsystem initialized [ 0.792189] ACPI: bus type USB registered [ 0.794124] usbcore: registered new interface driver usbfs [ 0.796106] usbcore: registered new interface driver hub [ 0.798100] usbcore: registered new device driver usb [ 0.800201] pps_core: LinuxPPS API ver. 1 registered [ 0.802012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.805062] PTP clock support registered [ 0.807038] EDAC MC: Ver: 3.0.0 [ 0.809115] PCI: Using ACPI for IRQ routing [ 0.811966] NetLabel: Initializing [ 0.813012] NetLabel: domain hash size = 128 [ 0.814009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.816068] NetLabel: unlabeled traffic allowed by default [ 0.818156] vgaarb: loaded [ 0.819287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.821015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.826368] clocksource: Switched to clocksource kvm-clock [ 0.935883] VFS: Disk quotas dquot_6.6.0 [ 0.937677] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.940369] *** VALIDATE ramfs *** [ 0.941782] *** VALIDATE hugetlbfs *** [ 0.943373] pnp: PnP ACPI init [ 0.945928] pnp: PnP ACPI: found 6 devices [ 0.965819] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.970220] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.972141] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.974173] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.976222] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.978668] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.981464] NET: Registered protocol family 2 [ 0.984094] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.988374] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.991541] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.996689] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.000764] TCP: Hash tables configured (established 65536 bind 65536) [ 1.003709] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.006345] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.009101] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013100] NET: Registered protocol family 1 [ 1.015873] RPC: Registered named UNIX socket transport module. [ 1.018019] RPC: Registered udp transport module. [ 1.019764] RPC: Registered tcp transport module. [ 1.021202] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.022506] NET: Registered protocol family 44 [ 1.024081] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.026287] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.028491] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.030807] PCI: CLS 0 bytes, default 64 [ 1.032256] Unpacking initramfs... [ 3.850679] debug: unmapping init [mem 0xffff8a67bcc54000-0xffff8a67bffbffff] [ 3.864455] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.871932] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.879084] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.754945] Initialise system trusted keyrings [ 4.757803] Key type blacklist registered [ 4.760116] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.783235] zbud: loaded [ 4.793326] *** VALIDATE nfs *** [ 4.799324] *** VALIDATE nfs4 *** [ 4.805650] pstore: using deflate compression [ 4.826305] Platform Keyring initialized [ 5.093426] NET: Registered protocol family 38 [ 5.096036] Key type asymmetric registered [ 5.099291] Asymmetric key parser 'x509' registered [ 5.105778] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.118525] io scheduler mq-deadline registered [ 5.122241] io scheduler kyber registered [ 5.130599] io scheduler bfq registered [ 5.138755] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.145332] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.153645] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.158424] ACPI: Power Button [PWRF] [ 5.167610] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.177399] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.211534] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.239171] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.276888] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.312785] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.354394] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.379227] Non-volatile memory driver v1.3 [ 5.383231] Linux agpgart interface v0.103 [ 5.464604] virtio_blk virtio1: [vda] 146024 512-byte logical blocks (74.8 MB/71.3 MiB) [ 5.475769] vda: detected capacity change from 0 to 74764288 [ 5.525657] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.537855] vdb: detected capacity change from 0 to 1073741824 [ 5.593588] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.599361] vdc: detected capacity change from 0 to 2621440000 [ 5.647933] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.658589] vdd: detected capacity change from 0 to 2621440000 [ 5.700923] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.704993] vde: detected capacity change from 0 to 4294967296 [ 5.736876] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.742272] vdf: detected capacity change from 0 to 4294967296 [ 5.761200] libphy: Fixed MDIO Bus: probed [ 5.770481] usbcore: registered new interface driver usbserial_generic [ 5.784632] usbserial: USB Serial support registered for generic [ 5.788871] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.809291] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.812742] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.821651] mousedev: PS/2 mouse device common for all mice [ 5.827334] rtc_cmos 00:05: RTC can wake from S4 [ 5.835622] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.838106] rtc_cmos 00:05: registered as rtc0 [ 5.845687] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.848384] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.853218] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.858932] intel_pstate: CPU model not supported [ 5.865641] hid: raw HID events driver (C) Jiri Kosina [ 5.882687] usbcore: registered new interface driver usbhid [ 5.885054] usbhid: USB HID core driver [ 5.889974] drop_monitor: Initializing network drop monitor service [ 5.895892] Initializing XFRM netlink socket [ 5.901626] NET: Registered protocol family 10 [ 5.904936] Segment Routing with IPv6 [ 5.906773] NET: Registered protocol family 17 [ 5.911419] mpls_gso: MPLS GSO support [ 5.924744] RAS: Correctable Errors collector initialized. [ 5.929990] AVX version of gcm_enc/dec engaged. [ 5.933370] AES CTR mode by8 optimization enabled [ 6.145392] sched_clock: Marking stable (6145362398, 0)->(7109686007, -964323609) [ 6.158408] registered taskstats version 1 [ 6.161402] Loading compiled-in X.509 certificates [ 6.165664] zswap: loaded using pool lzo/zbud [ 6.229870] Key type big_key registered [ 6.264678] Key type encrypted registered [ 6.267826] ima: No TPM chip found, activating TPM-bypass! [ 6.273242] ima: Allocated hash algorithm: sha1 [ 6.275452] ima: No architecture policies found [ 6.280388] evm: Initialising EVM extended attributes: [ 6.283878] evm: security.selinux [ 6.286168] evm: security.ima [ 6.287962] evm: security.capability [ 6.290146] evm: HMAC attrs: 0x1 [ 6.298262] rtc_cmos 00:05: setting system clock to 2026-09-02 00:40:03 UTC (1788309603) [ 6.311427] debug: unmapping init [mem 0xffffffffa5a03000-0xffffffffa5bfffff] [ 6.319322] debug: unmapping init [mem 0xffffffffa4782000-0xffffffffa4a58fff] [ 6.337072] Write protecting the kernel read-only data: 28672k [ 6.348838] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 6.354554] debug: unmapping init [mem 0xffffffffa3714000-0xffffffffa37fffff] [ 6.467524] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.491871] systemd[1]: Detected virtualization kvm. [ 6.496276] systemd[1]: Detected architecture x86-64. [ 6.498887] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.554193] systemd[1]: No hostname configured. [ 6.560167] systemd[1]: Set hostname to . [ 6.568235] random: systemd: uninitialized urandom read (16 bytes read) [ 6.577980] systemd[1]: Initializing machine ID from random generator. [ 6.656039] random: ln: uninitialized urandom read (6 bytes read) [ 6.931910] random: systemd: uninitialized urandom read (16 bytes read) [ 6.936802] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 6.956026] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.988339] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.446388] device-mapper: uevent: version 1.0.3 [ 8.449580] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.359659] virtio_net virtio0 ens2: renamed from eth0 [ 10.420841] random: fast init done [ 10.475656] scsi host0: ata_piix [ 10.649792] scsi host1: ata_piix [ 10.654810] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.663317] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.808204] random: crng init done [ 15.812566] random: 7 urandom warning(s) missed due to ratelimiting [ 16.886542] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.239183] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.913129] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.610802] SELinux: Disabled at runtime. [ 21.736599] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.751778] systemd[1]: Detected virtualization kvm. [ 21.754979] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.975418] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.981328] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.991338] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.000482] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.006874] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.023861] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.078357] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 23.477306] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 24.442386] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.427530] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.566637] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.804959] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.842458] EDAC sbridge: Ver: 1.1.2 [ 29.833138] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.667658] NFS: Registering the id_resolver key type [ 30.675187] Key type id_resolver registered [ 30.680145] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 90.125654] libcfs: loading out-of-tree module taints kernel. [ 90.149495] Key type ._llcrypt registered [ 90.150621] Key type .llcrypt registered [ 90.277073] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 108.842931] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 110.130364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.149909] alg: No test for adler32 (adler32-zlib) [ 111.536578] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 112.457675] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 114.312422] Key type lgssc registered [ 115.982301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.277509] hrtimer: interrupt took 5033475 ns [ 133.535083] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.204446] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 185.948781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.987486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.308693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.349839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.450289] Lustre: lustre-MDT0000: new disk, initializing [ 187.537832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.560132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.773523] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.292605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.384593] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.409837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.414504] Lustre: Skipped 1 previous similar message [ 204.486702] Lustre: lustre-MDT0001: new disk, initializing [ 204.548031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.575680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.585728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.547461] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.463818] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.823180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.140788] Lustre: lustre-OST0000: new disk, initializing [ 221.147067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.162304] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.238255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.538427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.555497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.646111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.775573] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.890981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.150581] Lustre: lustre-OST0001: new disk, initializing [ 242.159529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.173408] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.250367] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.406651] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.898122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.909199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.955479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.238182] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.790365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.977608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.853431] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 284.514754] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 287.048615] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 289.239792] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 290.780542] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 1 20:44:46 EDT 2026 [ 308.965741] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 310.407468] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.628774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.639079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.645477] Lustre: Skipped 3 previous similar messages [ 326.657979] Lustre: Skipped 3 previous similar messages [ 331.747530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.763641] Lustre: Skipped 3 previous similar messages [ 332.889802] Lustre: server umount lustre-MDT0000 complete [ 336.425628] LustreError: 7475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309933 with bad export cookie 6083381101626858418 [ 336.431967] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.440691] LustreError: 7475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 336.868453] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.876385] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.882287] Lustre: Skipped 1 previous similar message [ 336.890990] Lustre: Skipped 1 previous similar message [ 341.988863] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.990624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.010897] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 342.025149] Lustre: Skipped 1 previous similar message [ 342.868871] Lustre: server umount lustre-MDT0001 complete [ 346.281177] Lustre: server umount lustre-OST0000 complete [ 349.578411] Lustre: server umount lustre-OST0001 complete [ 355.960356] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 364.199390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 398.400259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.677965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.704793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.793543] Lustre: lustre-MDT0000: new disk, initializing [ 398.901716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.921549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.887271] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 415.179038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.293391] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.384736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.395528] Lustre: Skipped 1 previous similar message [ 415.503894] Lustre: lustre-MDT0001: new disk, initializing [ 415.681207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.740767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.760585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 420.993462] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 431.870305] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 437.751569] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 444.659846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.875193] Lustre: lustre-OST0000: new disk, initializing [ 444.878900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 444.885424] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 444.968487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.286071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.295214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 446.418461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.324190] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 459.976223] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 464.999656] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.230548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.796850] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.112249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.156757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.161616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.188810] Lustre: Skipped 1 previous similar message [ 487.392343] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 487.401713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.420644] Lustre: Skipped 4 previous similar messages [ 487.588699] Lustre: server umount lustre-OST0000 complete [ 492.514643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.516100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.535641] Lustre: Skipped 1 previous similar message [ 496.799909] Lustre: server umount lustre-MDT0000 complete [ 497.633600] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.654728] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 500.206171] LustreError: 15885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310097 with bad export cookie 6083381101626859951 [ 500.220586] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.223545] LustreError: 15885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.575459] Lustre: server umount lustre-MDT0001 complete [ 504.001151] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:48:19 (1788310099) [ 506.203113] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 507.784823] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:48:23 (1788310103) [ 513.470881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.859952] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.927087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.335654] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 519.139804] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.076664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.263892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 524.267995] Lustre: Skipped 2 previous similar messages [ 528.166031] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 534.903333] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.146322] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.976323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 552.189391] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 558.631395] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 574.956015] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.964988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.969227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 574.990588] Lustre: Skipped 1 previous similar message [ 578.303167] Lustre: server umount lustre-OST0000 complete [ 583.143314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.157146] Lustre: Skipped 1 previous similar message [ 588.614183] Lustre: server umount lustre-MDT0000 complete [ 591.744385] LustreError: 22263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310188 with bad export cookie 6083381101626860833 [ 591.756479] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.768481] LustreError: 22263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 592.134857] Lustre: server umount lustre-MDT0001 complete [ 600.662080] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 603.117225] Key type lgssc unregistered [ 603.322847] LNet: 23017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.330469] LNetError: 23017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.348427] LNet: Removed LNI 192.168.204.145@tcp [ 604.201162] Key type .llcrypt unregistered [ 604.202789] Key type ._llcrypt unregistered [ 621.253197] Key type ._llcrypt registered [ 621.257560] Key type .llcrypt registered [ 621.349805] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 622.067364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.179903] alg: No test for adler32 (adler32-zlib) [ 623.224329] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 623.452728] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 625.144508] Key type lgssc registered [ 626.032746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.472184] loop: module loaded [ 633.535779] loop0: detected capacity change from 0 to 204800000 [ 633.772053] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.782563] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.023975] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.715288] loop0: detected capacity change from 0 to 204800000 [ 639.903336] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.916085] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.927565] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.357301] loop0: detected capacity change from 0 to 204800000 [ 650.644428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 650.672285] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.920798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 651.977741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 652.076597] Lustre: lustre-MDT0000: new disk, initializing [ 652.177401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 652.204211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 656.171107] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 664.241141] loop1: detected capacity change from 0 to 204800000 [ 669.493259] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.743455] Lustre: lustre-OST07c6: new disk, initializing [ 669.753477] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 669.767483] Lustre: Skipped 1 previous similar message [ 669.781416] Lustre: 26182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 669.888877] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 671.315959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 671.334716] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 671.369412] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 675.791048] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 681.223580] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 681.232847] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 681.239207] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 681.245094] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 681.252518] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 681.258046] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 698.481382] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 698.486992] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.491416] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.496891] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 698.504039] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 698.509416] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.198967] Lustre: 26561:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 703.209866] Lustre: 26561:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.221300] Lustre: 26561:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 703.228483] Lustre: 26561:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.239075] Lustre: 26561:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.247582] Lustre: 26561:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.259590] Lustre: 26561:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 703.267238] Lustre: 26561:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.276377] Lustre: 26561:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.288174] Lustre: 26561:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.291939] Lustre: 26561:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.303164] Lustre: 26561:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.470679] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 706.480333] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 706.486910] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 706.494130] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.505788] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.512350] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.518798] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 706.523405] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.531252] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 706.542321] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.545803] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.551068] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.858804] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 710.874194] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 710.879737] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 710.884638] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.890806] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 710.896485] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.908368] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 710.917092] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.923317] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 710.929913] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 710.943779] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 710.951207] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.532657] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 720.541100] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 720.546559] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 720.552395] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.558547] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.564059] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.570706] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 720.577100] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.582235] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.590598] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.596155] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.601681] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 768.481985] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 768.493354] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.513686] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 773.608178] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 778.722559] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 782.304141] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 782.524691] Lustre: server umount lustre-OST07c6 complete [ 792.661133] Lustre: server umount lustre-MDT0000 complete [ 807.351678] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 809.737768] Key type lgssc unregistered [ 810.076995] LNet: 28742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.088160] LNetError: 28742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.135513] LNet: Removed LNI 192.168.204.145@tcp [ 810.990673] Key type .llcrypt unregistered [ 810.999140] Key type ._llcrypt unregistered [ 827.020186] Key type ._llcrypt registered [ 827.022762] Key type .llcrypt registered [ 827.119913] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 828.153968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 828.195666] alg: No test for adler32 (adler32-zlib) [ 829.272791] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 829.563107] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 831.256746] Key type lgssc registered [ 832.437166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.897243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.909869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.298438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.364288] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 849.890826] LustreError: 29994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.910834] LustreError: 29994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 852.234268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.703794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 856.704903] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 862.921584] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 867.484730] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 872.535208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.873688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.727857] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 884.264239] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 885.179743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 900.581113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.581608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.593804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.607770] Lustre: Skipped 1 previous similar message [ 905.318291] Lustre: server umount lustre-OST0000 complete [ 909.297879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 909.304657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.314337] Lustre: Skipped 1 previous similar message [ 909.320699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.817722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.819218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.285363] Lustre: server umount lustre-MDT0000 complete [ 915.945611] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.964870] LustreError: 29993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 918.451172] LustreError: 29974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310515 with bad export cookie 16164075659050685115 [ 918.462670] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.701800] Lustre: server umount lustre-MDT0001 complete [ 928.399668] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 931.032879] Key type lgssc unregistered [ 931.352323] LNet: 33737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.359738] LNetError: 33737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.374421] LNet: Removed LNI 192.168.204.145@tcp [ 932.179790] Key type .llcrypt unregistered [ 932.182322] Key type ._llcrypt unregistered [ 948.026939] Key type ._llcrypt registered [ 948.031322] Key type .llcrypt registered [ 948.139052] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 948.904746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.032192] alg: No test for adler32 (adler32-zlib) [ 950.072765] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 950.279927] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 951.968595] Key type lgssc registered [ 953.235748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.671771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.683094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.130185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.685986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 972.263805] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.290575] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 974.116314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.606363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.019242] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 985.655858] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 990.346269] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 996.560103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 996.939154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.037562] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1007.231163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1009.695864] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1019.891843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.898853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.925537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.949530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.951487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1022.978304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.454977] Lustre: server umount lustre-OST0000 complete [ 1031.141702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1031.151839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.164646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.176550] Lustre: Skipped 1 previous similar message [ 1033.186620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.187518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.507540] Lustre: server umount lustre-MDT0000 complete [ 1038.306061] LustreError: 34982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.326417] LustreError: 34982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1038.514373] LustreError: 35725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310635 with bad export cookie 16752048436327306047 [ 1038.523859] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1038.528563] LustreError: 35725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1038.807803] Lustre: server umount lustre-MDT0001 complete [ 1048.552271] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1051.298946] Key type lgssc unregistered [ 1051.569111] LNet: 38523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.573367] LNetError: 38523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.585464] LNet: Removed LNI 192.168.204.145@tcp [ 1052.463508] Key type .llcrypt unregistered [ 1052.468706] Key type ._llcrypt unregistered [ 1073.720199] Key type ._llcrypt registered [ 1073.723773] Key type .llcrypt registered [ 1073.823459] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1074.596738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1074.616828] alg: No test for adler32 (adler32-zlib) [ 1075.807749] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1076.128029] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1077.792348] Key type lgssc registered [ 1078.864794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1086.871775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1086.886054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.333348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.420830] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1098.728615] LustreError: 39950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.754260] LustreError: 39950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1102.476957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.057971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1107.903610] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1116.604225] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1122.467527] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1129.318260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1129.712308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.807265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1135.665616] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1142.599407] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1153.506414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.515540] Lustre: Skipped 2 previous similar messages [ 1153.516508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.528355] Lustre: Skipped 1 previous similar message [ 1154.743283] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1157.699797] Lustre: server umount lustre-MDT0000 complete [ 1158.629612] LustreError: 40677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.656591] LustreError: 40677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1159.828487] LustreError: 42441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.017686] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310758 with bad export cookie 17705452847807951272 [ 1161.023904] LustreError: 39930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1161.026421] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.746540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.747154] LustreError: 39944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.755942] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1166.754219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1166.760882] Lustre: Skipped 1 previous similar message [ 1167.543797] Lustre: server umount lustre-MDT0001 complete [ 1188.122255] Lustre: server umount lustre-OST0000 complete [ 1202.482615] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1204.973788] Key type lgssc unregistered [ 1205.211496] LNet: 43813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.223356] LNetError: 43813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.236100] LNet: Removed LNI 192.168.204.145@tcp [ 1205.934892] Key type .llcrypt unregistered [ 1205.938086] Key type ._llcrypt unregistered [ 1227.570658] Key type ._llcrypt registered [ 1227.572897] Key type .llcrypt registered [ 1227.679921] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1228.830229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.846326] alg: No test for adler32 (adler32-zlib) [ 1229.880393] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1230.188668] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1231.840375] Key type lgssc registered [ 1232.957271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.278389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.299953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.837808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.530294] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1251.303210] LustreError: 45231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.343787] LustreError: 45231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1255.332945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.659203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1260.061436] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1267.079567] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.381596] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1278.772543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1279.161152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.567741] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1286.318624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1293.255289] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1300.148202] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1300.962340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1300.978046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.996679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1301.987839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.999357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1302.015588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.043246] Lustre: Skipped 1 previous similar message [ 1304.956711] Lustre: server umount lustre-OST0000 complete [ 1321.520324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.524764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.536688] Lustre: Skipped 1 previous similar message [ 1321.545774] Lustre: Skipped 1 previous similar message [ 1326.562923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.567068] Lustre: Skipped 1 previous similar message [ 1327.960706] Lustre: server umount lustre-MDT0000 complete [ 1331.765852] LustreError: 45214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310928 with bad export cookie 15323029464446762984 [ 1331.788647] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1331.797094] LustreError: 45214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1338.196818] Lustre: server umount lustre-MDT0001 complete [ 1348.621725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1351.700678] Key type lgssc unregistered [ 1352.034538] LNet: 48956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.051761] LNetError: 48956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.106700] LNet: Removed LNI 192.168.204.145@tcp [ 1353.127749] Key type .llcrypt unregistered [ 1353.136496] Key type ._llcrypt unregistered [ 1375.861881] Key type ._llcrypt registered [ 1375.865948] Key type .llcrypt registered [ 1376.023954] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1377.015762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.026584] alg: No test for adler32 (adler32-zlib) [ 1378.038731] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1378.242867] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1379.904231] Key type lgssc registered [ 1380.951878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.114659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.131342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.603599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.734752] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1400.813434] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.833878] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1403.778898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.157752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1408.428911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1416.320992] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1423.197934] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1430.335678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.631631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.698347] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1436.792853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1444.288150] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1453.979198] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1457.109548] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1469.360730] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0121h04m24s [ 1471.415210] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1475.047435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.053347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.068311] Lustre: Skipped 2 previous similar messages [ 1475.089318] Lustre: Skipped 2 previous similar messages [ 1476.141605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.152573] Lustre: Skipped 2 previous similar messages [ 1477.797403] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1480.092411] Lustre: server umount lustre-MDT0000 complete [ 1482.218246] LustreError: 52866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.242489] LustreError: 52866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1483.271687] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1484.954572] LustreError: 50370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.361502] LustreError: 51409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.383098] LustreError: 51409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1490.152763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.326174] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.677027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1490.757296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1491.709017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1495.774832] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1498.945953] Lustre: DEBUG MARKER: Wait for df (36706) ... [ 1500.482473] Lustre: DEBUG MARKER: done [ 1505.767644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1505.778403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.804040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1505.811397] Lustre: Skipped 1 previous similar message [ 1506.275515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.663108] Lustre: server umount lustre-OST0000 complete [ 1517.537894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.540699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1517.545939] Lustre: Skipped 1 previous similar message [ 1517.558598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.575782] Lustre: Skipped 4 previous similar messages [ 1520.370674] Lustre: server umount lustre-MDT0000 complete [ 1523.675562] LustreError: 50356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311120 with bad export cookie 493005253553214671 [ 1523.679568] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.696229] LustreError: 50356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1524.072261] Lustre: server umount lustre-MDT0001 complete [ 1533.536625] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1536.274828] Key type lgssc unregistered [ 1536.511632] LNet: 55302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.516702] LNetError: 55302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.533227] LNet: Removed LNI 192.168.204.145@tcp [ 1537.256635] Key type .llcrypt unregistered [ 1537.260077] Key type ._llcrypt unregistered [ 1576.053847] Key type ._llcrypt registered [ 1576.057665] Key type .llcrypt registered [ 1576.187835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1577.642718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1577.684606] alg: No test for adler32 (adler32-zlib) [ 1578.759387] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1579.019642] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1580.704193] Key type lgssc registered [ 1581.848823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.339387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.365611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.653641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1590.684496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1590.834190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.649873] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1604.782747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.925944] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1604.980153] Lustre: 57513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1605.233812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.997256] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1617.855822] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1623.999277] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1631.015836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1631.259991] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1631.464834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.153163] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1638.568925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1644.896079] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.824338] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0121h07m27s [ 1653.341500] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1660.774279] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1663.872555] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1663.929712] LustreError: 58340:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1684.175637] Lustre: lustre-MDT0000: Client 3ef5ddf8-026e-498c-b87c-ff7115c3806e (at 192.168.204.45@tcp) reconnecting [ 1703.960441] LustreError: 58340:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1706.183313] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1708.575844] Lustre: DEBUG MARKER: done [ 1715.683235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1715.688178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.706277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.709965] Lustre: Skipped 1 previous similar message [ 1715.721455] Lustre: Skipped 1 previous similar message [ 1720.035767] Lustre: server umount lustre-OST0000 complete [ 1725.924141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.932718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.956359] Lustre: Skipped 1 previous similar message [ 1725.981937] Lustre: Skipped 1 previous similar message [ 1727.012462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.024478] Lustre: Skipped 1 previous similar message [ 1733.091179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1733.102852] Lustre: Skipped 1 previous similar message [ 1737.203363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.222293] Lustre: Skipped 1 previous similar message [ 1738.208159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1738.377108] Lustre: server umount lustre-MDT0000 complete [ 1742.650988] LustreError: 57505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311339 with bad export cookie 15894917736779840147 [ 1742.654496] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1742.668605] LustreError: 57505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1743.173548] Lustre: server umount lustre-MDT0001 complete [ 1755.149350] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1758.579898] Key type lgssc unregistered [ 1758.954934] LNet: 61848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.965354] LNetError: 61848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.989677] LNet: Removed LNI 192.168.204.145@tcp [ 1759.874524] Key type .llcrypt unregistered [ 1759.878449] Key type ._llcrypt unregistered [ 1804.212254] Key type ._llcrypt registered [ 1804.215306] Key type .llcrypt registered [ 1804.303712] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1805.071494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.087798] alg: No test for adler32 (adler32-zlib) [ 1806.229533] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1806.526580] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1808.249845] Key type lgssc registered [ 1809.400851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.889943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1812.904679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.231051] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1814.253300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1814.453824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.409050] Lustre: Failing over lustre-MDT0000 [ 1815.845927] Lustre: server umount lustre-MDT0000 complete [ 1822.922503] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:10:18 (1788311418) [ 1830.012056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.635632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.968423] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1845.162708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.277916] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1845.349754] Lustre: 64646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1845.662922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1850.804448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1858.677834] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1864.564429] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1870.812894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1871.116555] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1871.442514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.095772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1877.341567] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1883.341725] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.508537] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1896.930850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1896.945850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.961218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.445685] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.465564] Lustre: Skipped 1 previous similar message [ 1897.471495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.475328] Lustre: Skipped 2 previous similar messages [ 1900.853346] Lustre: server umount lustre-MDT0000 complete [ 1906.123378] LustreError: 64639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311503 with bad export cookie 11940561277863892294 [ 1906.130508] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1906.172344] LustreError: 64639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1906.666706] Lustre: server umount lustre-MDT0001 complete [ 1909.648719] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1914.301860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1921.657466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.040340] Lustre: 63573:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311504/real 1788311504] req@000000003d9f42f2 x1875180417752832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788311520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1923.055121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.136201] Lustre: 63573:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311508/real 1788311508] req@00000000b33406df x1875180417752960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788311524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1928.160364] Lustre: 63571:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311509/real 1788311509] req@000000000751813d x1875180417753088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788311525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1931.314272] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa5b56537e9dc9e29 [ 1931.336895] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [ 1931.702573] LustreError: 66496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.725273] LustreError: 66496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1931.860299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.965623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1932.832232] Lustre: 63571:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311513/real 1788311513] req@00000000aa2969d1 x1875180417753344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788311529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1932.841845] LustreError: 68418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.877567] LustreError: 68418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1935.824574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1937.888260] Lustre: 63572:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788311518/real 1788311518] req@000000001931e47a x1875180417753472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788311534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.892615] LustreError: 68415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.962358] LustreError: 68415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1943.016467] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.037313] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1943.811942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.262539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1948.828552] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1949.687833] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1952.809844] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1956.221914] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1961.565662] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1968.110131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.121856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1968.123600] Lustre: Skipped 2 previous similar messages [ 1968.135696] Lustre: Skipped 2 previous similar messages [ 1968.298986] Lustre: server umount lustre-MDT0000 complete [ 1971.858162] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311569 with bad export cookie 11940561277863894569 [ 1971.860586] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1971.869483] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1973.217950] LustreError: 69906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.220559] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.267602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1974.342757] Lustre: server umount lustre-MDT0001 complete [ 1980.555833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1989.592578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.728725] LustreError: 63569:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000090256afc x1875180417779328/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1998.060420] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.127588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.176646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2002.671515] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2011.247121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.648297] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2015.923696] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2020.843249] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2024.157738] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.747749] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2037.218641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2037.227231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.251132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.170422] Lustre: server umount lustre-MDT0000 complete [ 2046.433774] LustreError: 71101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.454703] LustreError: 71101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2046.825799] LustreError: 64638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311644 with bad export cookie 11940561277863895948 [ 2046.834864] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.841712] LustreError: 64638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.173996] Lustre: server umount lustre-MDT0001 complete [ 2053.532320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2056.788384] Lustre: DEBUG MARKER: files compared the same [ 2069.301945] Lustre: server umount lustre-OST0000 complete [ 2086.096877] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2088.377466] Key type lgssc unregistered [ 2088.676462] LNet: 74382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.682568] LNetError: 74382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.705076] LNet: Removed LNI 192.168.204.145@tcp [ 2089.625364] Key type .llcrypt unregistered [ 2089.627222] Key type ._llcrypt unregistered [ 2111.652153] Key type ._llcrypt registered [ 2111.654602] Key type .llcrypt registered [ 2111.757907] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2112.819898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2112.993027] alg: No test for adler32 (adler32-zlib) [ 2114.081633] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2114.315855] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2115.968171] Key type lgssc registered [ 2117.104991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2124.909612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2124.929976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.364906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.526559] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2136.553477] LustreError: 75798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.573980] LustreError: 75798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2137.779274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.967383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.461487] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2147.423730] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.934096] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2157.150911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2157.393083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.662440] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2168.946336] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2171.761547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2182.113991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.119883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.124333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.137292] Lustre: Skipped 1 previous similar message [ 2186.414239] Lustre: server umount lustre-OST0000 complete [ 2192.356194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2192.384140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.396065] Lustre: Skipped 2 previous similar messages [ 2196.310550] Lustre: server umount lustre-MDT0000 complete [ 2197.478446] LustreError: 75797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.492799] LustreError: 75797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2200.187302] LustreError: 75777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311797 with bad export cookie 16219548133438781430 [ 2200.197750] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.202229] LustreError: 75777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2200.577372] Lustre: server umount lustre-MDT0001 complete [ 2212.977337] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2215.828249] Key type lgssc unregistered [ 2216.401908] LNet: 79332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.418902] LNetError: 79332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.438701] LNet: Removed LNI 192.168.204.145@tcp [ 2217.932075] Key type .llcrypt unregistered [ 2217.934458] Key type ._llcrypt unregistered [ 2245.551873] Key type ._llcrypt registered [ 2245.556786] Key type .llcrypt registered [ 2245.686099] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2246.580175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.634810] alg: No test for adler32 (adler32-zlib) [ 2247.877239] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2248.126889] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2249.796609] Key type lgssc registered [ 2251.046648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.726695] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.742542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.402522] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2279.520341] LustreError: 80725:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 2306.272234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.350141] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2319.813902] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2326.818116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.998749] Lustre: *** cfs_fail_loc=706, val=0*** [ 2327.443393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.523823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2332.650776] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.674390] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2332.738278] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2337.765408] LustreError: 80751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.838178] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.864520] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2341.922065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.342786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.714432] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2353.809978] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2359.703833] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2362.855144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.855432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2362.884739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.901874] Lustre: Skipped 1 previous similar message [ 2367.634688] Lustre: server umount lustre-OST0000 complete [ 2373.090963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.093788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2373.124689] Lustre: Skipped 1 previous similar message [ 2373.131837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.152341] Lustre: Skipped 1 previous similar message [ 2378.020393] Lustre: server umount lustre-MDT0000 complete [ 2378.212938] LustreError: 81814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.221234] LustreError: 81814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2382.331946] LustreError: 80732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311979 with bad export cookie 17149887495529300501 [ 2382.336948] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.346872] LustreError: 80732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2382.568445] Lustre: server umount lustre-MDT0001 complete [ 2394.421488] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2396.910768] Key type lgssc unregistered [ 2397.191549] LNet: 84343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.199771] LNetError: 84343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.212886] LNet: Removed LNI 192.168.204.145@tcp [ 2397.812624] Key type .llcrypt unregistered [ 2397.815245] Key type ._llcrypt unregistered [ 2423.468938] Key type ._llcrypt registered [ 2423.471307] Key type .llcrypt registered [ 2423.611543] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2425.079413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.129645] alg: No test for adler32 (adler32-zlib) [ 2426.302954] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2426.574894] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2428.296395] Key type lgssc registered [ 2429.659832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2437.751783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2437.778712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.312736] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2452.355340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.937482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.183258] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2458.082387] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.097843] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2463.203594] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.223238] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2464.890431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.137866] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.168699] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2465.196159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.373538] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.388986] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2470.885826] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2475.492801] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.511350] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2479.189703] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.206217] Lustre: MGS: Client beb8d625-bb9e-45f8-9f70-54fcefc7b083 (at 0@lo) reconnecting [ 2479.232600] LustreError: 86461:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2479.479241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.588809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2484.510458] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2500.077345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.096463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.098114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2500.116084] Lustre: Skipped 1 previous similar message [ 2502.788328] Lustre: server umount lustre-OST0000 complete [ 2510.309120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2510.313213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.319078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2510.331081] Lustre: Skipped 2 previous similar messages [ 2512.478910] Lustre: server umount lustre-MDT0000 complete [ 2515.425610] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.439896] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2516.255456] Lustre: server umount lustre-MDT0001 complete [ 2519.590395] Lustre: server umount MGS complete [ 2529.956356] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:22:05 (1788312125) [ 2556.801985] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2566.676128] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2602.209728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.210446] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2615.420945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.624992] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2615.687442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2615.745476] Lustre: lustre-MDT0001: new disk, initializing [ 2615.820283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2615.851454] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2615.863584] Lustre: 93615:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2615.872387] LustreError: 93615:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2618.912179] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2627.724393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2627.939613] Lustre: lustre-OST0000: new disk, initializing [ 2627.942773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2627.949679] Lustre: 94380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2628.056560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.616085] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2635.374756] Lustre: 94704:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fd228a94 x1875181067869568/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788312248 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2644.334924] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2650.412223] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.421482] Lustre: MGS: Client ab974c61-0c74-48fe-97eb-9164957aa222 (at 0@lo) reconnecting [ 2650.448931] Lustre: 92884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2650.466295] LustreError: 93601:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2650.508616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2650.578682] Lustre: lustre-MDT0000: new disk, initializing [ 2650.709627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.742879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2650.770321] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2651.818098] Lustre: 95450:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000013aee6de x1875181067876480/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788312265 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2655.647400] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2657.015202] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2660.977981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2660.986725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2661.067685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2686.948479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2686.957572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.968892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2686.974102] Lustre: Skipped 1 previous similar message [ 2688.999306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.005872] Lustre: Skipped 1 previous similar message [ 2694.119139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.131018] Lustre: Skipped 1 previous similar message [ 2700.256244] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2700.423354] Lustre: server umount lustre-OST0000 complete [ 2709.475347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.483516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.488781] LustreError: Skipped 1 previous similar message [ 2709.497685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.506710] Lustre: Skipped 2 previous similar messages [ 2709.524641] Lustre: Skipped 3 previous similar messages [ 2710.651257] Lustre: server umount lustre-MDT0000 complete [ 2720.832524] Lustre: server umount lustre-MDT0001 complete [ 2724.919421] Lustre: server umount MGS complete [ 2735.845512] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:25:31 (1788312331) [ 2757.368645] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2760.617095] Key type lgssc unregistered [ 2760.881827] LNet: 97956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2760.888992] LNetError: 97956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2760.904513] LNet: Removed LNI 192.168.204.145@tcp [ 2761.953181] Key type .llcrypt unregistered [ 2761.955244] Key type ._llcrypt unregistered [ 2779.934919] Key type ._llcrypt registered [ 2779.936733] Key type .llcrypt registered [ 2780.031164] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2780.752741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.771757] alg: No test for adler32 (adler32-zlib) [ 2781.946440] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2782.263798] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2783.922627] Key type lgssc registered [ 2784.995451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2791.916495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2791.931804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.368213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.412975] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2803.684930] LustreError: 99178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.710731] LustreError: 99178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2807.243504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.664566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2811.792677] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2820.142615] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2825.708419] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2832.694254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2833.388978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.519085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2840.583139] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2847.771409] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2858.984700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.997968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.017725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.053916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.064865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.069859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.170948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.183060] Lustre: Skipped 2 previous similar messages [ 2872.293651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.304663] Lustre: Skipped 1 previous similar message [ 2873.312192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2873.625408] Lustre: server umount lustre-OST0000 complete [ 2877.411889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.439382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2883.780778] Lustre: server umount lustre-MDT0000 complete [ 2887.512403] LustreError: 99159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312484 with bad export cookie 4194989173954550690 [ 2887.514714] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.529532] LustreError: 99159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2887.862823] Lustre: server umount lustre-MDT0001 complete [ 2898.668262] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2901.486908] Key type lgssc unregistered [ 2901.814135] LNet: 102717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2901.821862] LNetError: 102717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2901.833966] LNet: Removed LNI 192.168.204.145@tcp [ 2902.679509] Key type .llcrypt unregistered [ 2902.681282] Key type ._llcrypt unregistered [ 2917.859167] Key type ._llcrypt registered [ 2917.863876] Key type .llcrypt registered [ 2917.962231] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2919.587665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2919.637293] alg: No test for adler32 (adler32-zlib) [ 2920.823474] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2921.183597] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2922.968283] Key type lgssc registered [ 2924.482728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2931.088558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2931.097557] LustreError: 103793:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2931.138293] LustreError: 103803:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2931.144576] LustreError: 103793:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4964 [ 2931.152109] LustreError: 103793:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2931.161923] LustreError: 103793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2931.191714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.700107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.987439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.460600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.857859] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2952.778595] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lsmod [ 2955.891088] LustreError: 105002:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2955.950872] LustreError: 105009:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2955.956629] LustreError: 105002:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4942 [ 2955.962686] LustreError: 105002:0:(obd_mount.c:234:lustre_start_simple()) lustre-OST0000-osd attach error -17 [ 2955.985159] LustreError: 105002:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -17 [ 2955.997077] LustreError: 105002:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 2956.004655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.518919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.574188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2961.890947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2961.898143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.908519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.916076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.919275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.944393] Lustre: Skipped 1 previous similar message [ 2965.752842] Lustre: server umount lustre-MDT0000 complete [ 2966.053122] LustreError: 103837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.063234] LustreError: 103837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2969.299182] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312566 with bad export cookie 12648622908242806979 [ 2969.302592] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2969.307894] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2970.209043] LustreError: 104272:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c8c96ba0 x1875181586895872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2970.461775] Lustre: server umount lustre-MDT0001 complete [ 2984.805502] Lustre: server umount lustre-OST0000 complete [ 2993.801205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.212447] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.276114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.267928] LustreError: 106074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.352992] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3004.387370] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.480447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3008.878597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3014.038403] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3022.035840] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3028.266531] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3034.805192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.223442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.347447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3041.765339] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3049.142455] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3060.195898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3060.208074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.225810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.233689] Lustre: Skipped 2 previous similar messages [ 3062.177261] Lustre: server umount lustre-OST0000 complete [ 3069.416067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.421021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.429352] Lustre: Skipped 1 previous similar message [ 3069.444502] Lustre: Skipped 1 previous similar message [ 3071.934592] Lustre: server umount lustre-MDT0000 complete [ 3074.535214] LustreError: 106074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.567209] LustreError: 106074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3075.644853] LustreError: 106054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312672 with bad export cookie 12648622908242807917 [ 3075.649403] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3075.658390] LustreError: 106054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3085.560745] Lustre: server umount lustre-MDT0001 complete [ 3097.349898] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3100.100334] Key type lgssc unregistered [ 3100.476623] LNet: 109619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3100.481959] LNetError: 109619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3100.494546] LNet: Removed LNI 192.168.204.145@tcp [ 3101.562154] Key type .llcrypt unregistered [ 3101.564705] Key type ._llcrypt unregistered [ 3125.966464] Key type ._llcrypt registered [ 3125.968826] Key type .llcrypt registered [ 3126.063158] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3127.033427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3127.169850] alg: No test for adler32 (adler32-zlib) [ 3128.294135] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3128.514911] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3130.248259] Key type lgssc registered [ 3131.588795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3139.795332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3139.824143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.206208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.273576] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3151.335749] LustreError: 111035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.381456] LustreError: 111035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3155.965489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.373866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.819399] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3170.444799] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3176.829838] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3184.289150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.720951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.819456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3193.010271] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3202.909782] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3208.450437] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3213.183856] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3217.894084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3217.905058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.922796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3221.986515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3221.987574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3221.991924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.324754] Lustre: server umount lustre-OST0000 complete [ 3227.109238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.114821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.129283] Lustre: Skipped 1 previous similar message [ 3227.134442] Lustre: Skipped 1 previous similar message [ 3232.228511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.234856] Lustre: Skipped 2 previous similar messages [ 3233.343610] Lustre: server umount lustre-MDT0000 complete [ 3236.881401] LustreError: 111017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312834 with bad export cookie 13327454192746358829 [ 3236.883538] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.896890] LustreError: 111017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3237.176151] Lustre: server umount lustre-MDT0001 complete [ 3247.165413] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3249.986689] Key type lgssc unregistered [ 3250.299229] LNet: 114677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3250.306100] LNetError: 114677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3250.320911] LNet: Removed LNI 192.168.204.145@tcp [ 3251.200920] Key type .llcrypt unregistered [ 3251.206400] Key type ._llcrypt unregistered [ 3268.686866] Key type ._llcrypt registered [ 3268.689436] Key type .llcrypt registered [ 3268.805244] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3269.920691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3270.139938] alg: No test for adler32 (adler32-zlib) [ 3271.242865] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3271.509071] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3273.184216] Key type lgssc registered [ 3274.443820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3282.412245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3282.427724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.895649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.441996] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3294.189250] LustreError: 115900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3294.207199] LustreError: 115900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3297.109066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.510547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3302.542803] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3309.818140] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3315.204839] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3324.611043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.230671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.371696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3332.291611] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3340.073322] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3353.057866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.065475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.077124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.088868] Lustre: Skipped 1 previous similar message [ 3358.102830] Lustre: server umount lustre-OST0000 complete [ 3363.300217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.332432] Lustre: Skipped 1 previous similar message [ 3363.342714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.419805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.431384] Lustre: Skipped 2 previous similar messages [ 3368.694674] Lustre: server umount lustre-MDT0000 complete [ 3372.868654] LustreError: 115881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312970 with bad export cookie 7958628299273949943 [ 3372.882746] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.895126] LustreError: 115881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3373.229978] Lustre: server umount lustre-MDT0001 complete [ 3385.547335] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3388.814276] Key type lgssc unregistered [ 3389.223265] LNet: 119439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3389.237504] LNetError: 119439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3389.262064] LNet: Removed LNI 192.168.204.145@tcp [ 3390.261773] Key type .llcrypt unregistered [ 3390.270612] Key type ._llcrypt unregistered [ 3413.524398] Key type ._llcrypt registered [ 3413.526547] Key type .llcrypt registered [ 3413.634701] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3414.384039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3414.484887] alg: No test for adler32 (adler32-zlib) [ 3415.541856] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3415.780127] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3417.424146] Key type lgssc registered [ 3418.525741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3426.506697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.539421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.053792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.739371] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3438.580224] LustreError: 120856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.602649] LustreError: 120856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3441.822886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.267125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3446.753292] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3454.307901] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3460.285945] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.364934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.966145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.074644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3475.676688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3483.392476] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3493.376433] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3497.226810] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3497.232383] Lustre: Skipped 2 previous similar messages [ 3501.641801] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3501.672759] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3501.815894] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3501.823870] Lustre: Skipped 2 previous similar messages [ 3509.343407] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3509.348851] Lustre: Skipped 1 previous similar message [ 3514.486882] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3514.495333] Lustre: Skipped 2 previous similar messages [ 3517.707068] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp [ 3523.933585] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.45/24@tcp in log lustre-MDT0000 [ 3523.942180] Lustre: Skipped 2 previous similar messages [ 3525.556520] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3525.565485] Lustre: Skipped 1 previous similar message [ 3532.776070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3532.777120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.786393] LustreError: Skipped 1 previous similar message [ 3532.797679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.797509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.821487] LustreError: 122900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.849992] LustreError: 122900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3533.981344] Lustre: server umount lustre-OST0000 complete [ 3538.916041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.928923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.934605] Lustre: Skipped 1 previous similar message [ 3538.944826] Lustre: Skipped 1 previous similar message [ 3544.013370] Lustre: server umount lustre-MDT0000 complete [ 3544.040067] LustreError: 121865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.060016] LustreError: 121865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3547.524663] LustreError: 120838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313144 with bad export cookie 15039898990317578823 [ 3547.527230] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.543537] LustreError: 120838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3547.943728] Lustre: server umount lustre-MDT0001 complete [ 3558.404675] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3561.294348] Key type lgssc unregistered [ 3561.662366] LNet: 125754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.681281] LNetError: 125754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.712071] LNet: Removed LNI 192.168.204.145@tcp [ 3562.693602] Key type .llcrypt unregistered [ 3562.695658] Key type ._llcrypt unregistered [ 3592.904261] Key type ._llcrypt registered [ 3592.907939] Key type .llcrypt registered [ 3593.022600] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3594.333138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3594.379750] alg: No test for adler32 (adler32-zlib) [ 3595.483890] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3595.783092] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3597.488192] Key type lgssc registered [ 3598.442452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3606.406635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3606.431504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.978555] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3607.997511] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.45/24@tcp [ 3608.030505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.280637] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3618.276568] LustreError: 127313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3618.297646] LustreError: 127313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3623.395103] LustreError: 127314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.645182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.877450] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3623.907387] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3623.970975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3628.994608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3638.728261] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3645.852575] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3653.260958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.619476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.745051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3661.371338] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3668.365933] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3679.204277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3679.216381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.224418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3679.245526] Lustre: Skipped 1 previous similar message [ 3679.262676] Lustre: Skipped 1 previous similar message [ 3683.851642] Lustre: server umount lustre-OST0000 complete [ 3689.444256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.452677] Lustre: Skipped 1 previous similar message [ 3689.453936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.465115] Lustre: Skipped 1 previous similar message [ 3693.436599] Lustre: server umount lustre-MDT0000 complete [ 3694.564725] LustreError: 127310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.588265] LustreError: 127310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3697.217283] LustreError: 127296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313294 with bad export cookie 1149369182111970159 [ 3697.233085] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.249020] LustreError: 127296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3697.523443] Lustre: server umount lustre-MDT0001 complete [ 3709.456270] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3712.106926] Key type lgssc unregistered [ 3712.434458] LNet: 130906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.442979] LNetError: 130906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3713.509844] LNet: Removed LNI 192.168.204.145@tcp [ 3714.466424] Key type .llcrypt unregistered [ 3714.472234] Key type ._llcrypt unregistered [ 3743.287390] Key type ._llcrypt registered [ 3743.293648] Key type .llcrypt registered [ 3743.430948] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3757.813201] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3758.903690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3758.931337] alg: No test for adler32 (adler32-zlib) [ 3760.009179] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3760.257375] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3761.944883] Key type lgssc registered [ 3763.150714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3797.747122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3797.809714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.989990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3799.051874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3799.176606] Lustre: lustre-MDT0000: new disk, initializing [ 3799.269884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.292127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3804.342212] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3816.711931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.861242] Lustre: 134308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3816.961348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3816.967336] Lustre: Skipped 1 previous similar message [ 3817.101671] Lustre: lustre-MDT0001: new disk, initializing [ 3817.225644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3817.288339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3817.305062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3823.010829] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3832.637629] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3838.442214] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3844.782418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3845.047573] Lustre: lustre-OST0000: new disk, initializing [ 3845.053558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3845.060427] Lustre: 136279:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3845.143821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.751128] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3852.898530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3852.909821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3853.024306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3860.210562] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3864.897212] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3865.159398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3869.837961] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3870.023635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3873.761438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.770415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.786476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3873.799093] Lustre: Skipped 1 previous similar message [ 3878.885942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3878.898529] Lustre: Skipped 1 previous similar message [ 3884.006957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.014301] Lustre: Skipped 1 previous similar message [ 3886.048190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3886.177894] Lustre: server umount lustre-OST0000 complete [ 3892.196406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.202282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3892.211122] Lustre: Skipped 2 previous similar messages [ 3892.220163] Lustre: Skipped 1 previous similar message [ 3895.612270] Lustre: server umount lustre-MDT0000 complete [ 3898.823969] LustreError: 136278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313496 with bad export cookie 15579434462690917192 [ 3898.826021] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.856307] LustreError: 136278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3899.205618] Lustre: server umount lustre-MDT0001 complete [ 3909.466028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.981446] LustreError: 138493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.005595] LustreError: 138493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3910.078179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.257100] LustreError: 138494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.304444] LustreError: 138494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3916.602861] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3920.365571] LustreError: 138493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3925.309393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.480353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3925.591340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3930.064777] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3936.763995] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3941.817326] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3947.935324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3948.209097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.146651] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3961.572411] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.048606] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3966.347599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3970.632760] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3970.823603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3980.396670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3980.555277] Lustre: lustre-OST0001: new disk, initializing [ 3980.558616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3980.564232] Lustre: 141865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3980.638399] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.629310] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3990.623820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3990.642798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3990.750829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3995.655451] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3995.932686] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4000.827735] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4001.079803] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4006.911455] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99498837a800.ost_server_uuid 50 [ 4008.283302] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99498837a800.ost_server_uuid in IDLE state after 0 sec [ 4019.212793] Lustre: server umount lustre-OST0001 complete [ 4021.745304] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.746503] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4021.769587] LustreError: 141517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.770747] LustreError: Skipped 2 previous similar messages [ 4021.808804] LustreError: 141517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4026.855481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4026.861481] LustreError: 141517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.867046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.889357] Lustre: Skipped 2 previous similar messages [ 4026.935400] LustreError: 141517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4029.211289] Lustre: server umount lustre-OST0000 complete [ 4037.096116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.105546] Lustre: Skipped 1 previous similar message [ 4039.703815] Lustre: server umount lustre-MDT0000 complete [ 4042.217264] LustreError: 139498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.241622] LustreError: 139498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4043.670847] LustreError: 141863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313640 with bad export cookie 15579434462690918074 [ 4043.679240] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.688033] LustreError: 141863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4044.220337] Lustre: server umount lustre-MDT0001 complete [ 4062.935859] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4065.906537] Key type lgssc unregistered [ 4066.244577] LNet: 144872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4066.253279] LNetError: 144872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4066.266260] LNet: Removed LNI 192.168.204.145@tcp [ 4067.317984] Key type .llcrypt unregistered [ 4067.325513] Key type ._llcrypt unregistered [ 4110.081793] Key type ._llcrypt registered [ 4110.085938] Key type .llcrypt registered [ 4110.248299] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4126.941471] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4128.391342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.459146] alg: No test for adler32 (adler32-zlib) [ 4129.524653] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 4129.762134] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4131.480299] Key type lgssc registered [ 4132.707475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4169.241828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4169.288243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.566295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4170.601089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4170.711523] Lustre: lustre-MDT0000: new disk, initializing [ 4170.802619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.827335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4175.311048] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4186.230679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.410424] Lustre: 149031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4186.547157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4186.558724] Lustre: Skipped 1 previous similar message [ 4186.687346] Lustre: lustre-MDT0001: new disk, initializing [ 4186.792239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4186.849906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4186.860364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4191.822757] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4202.550597] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4209.416390] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4217.485702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4217.781926] Lustre: lustre-OST0000: new disk, initializing [ 4217.802790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4217.816283] Lustre: 150997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4217.955446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.281927] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4227.202780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4227.217721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4227.356795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4235.480913] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4241.039626] Lustre: Failing over lustre-OST0000 [ 4242.916151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4242.918685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.924344] LustreError: Skipped 1 previous similar message [ 4242.934880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4242.950105] Lustre: Skipped 1 previous similar message [ 4243.281972] Lustre: server umount lustre-OST0000 complete [ 4263.481116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4264.019898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.041976] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4269.214050] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4271.173217] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4275.369713] Lustre: Failing over lustre-MDT0000 [ 4275.676870] Lustre: server umount lustre-MDT0000 complete [ 4279.265569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4279.266112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.285106] Lustre: Skipped 2 previous similar messages [ 4284.563686] LustreError: 149038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.580326] LustreError: 149038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4288.490494] LustreError: 149042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.512269] LustreError: 149042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4289.680524] LustreError: 149037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.601664] LustreError: 151224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.584125] Lustre: 147251:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788313876/real 1788313876] req@0000000071787ad3 x1875182853599616/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788313892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4295.603446] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4296.604093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.773082] LustreError: 149038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.786116] LustreError: 149038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4304.864947] LustreError: 147250:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008eddf3f5 x1875182853603968/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4305.156335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4305.275724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.353708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4306.069712] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4310.186563] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4310.521326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4310.575168] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4310.641109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4320.741643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.775860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4320.781791] Lustre: Skipped 2 previous similar messages [ 4324.804544] Lustre: server umount lustre-OST0000 complete [ 4330.982164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.983174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4330.990060] Lustre: Skipped 2 previous similar messages [ 4331.006789] Lustre: Skipped 1 previous similar message [ 4334.802741] Lustre: server umount lustre-MDT0000 complete [ 4336.100639] LustreError: 149041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.110847] LustreError: 149041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4339.416499] LustreError: 149022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313936 with bad export cookie 4308099499456925092 [ 4339.425174] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4339.431045] LustreError: 149022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4339.843964] Lustre: server umount lustre-MDT0001 complete [ 4352.559651] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4355.941590] Key type lgssc unregistered [ 4356.282463] LNet: 154658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4356.290128] LNetError: 154658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4356.314043] LNet: Removed LNI 192.168.204.145@tcp [ 4357.219547] Key type .llcrypt unregistered [ 4357.222747] Key type ._llcrypt unregistered [ 4390.715653] Key type ._llcrypt registered [ 4390.724481] Key type .llcrypt registered [ 4390.810170] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4391.519047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4391.573671] alg: No test for adler32 (adler32-zlib) [ 4392.570658] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 4392.746390] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4394.400208] Key type lgssc registered [ 4395.358538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4403.468880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4403.486844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.125704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4410.954885] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4415.465652] LustreError: 156181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.472995] LustreError: 156181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4420.604260] LustreError: 156180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.016562] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4424.525384] LustreError: 156209:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4424.535545] LustreError: 156209:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4424.541562] LustreError: 156209:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4424.894293] Lustre: server umount lustre-MDT0000 complete [ 4441.653858] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4454.172642] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4501.935870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4502.245818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4502.271665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4502.335303] Lustre: lustre-MDT0000: new disk, initializing [ 4502.450950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4502.470802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4507.947329] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4519.731245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.890866] Lustre: 159515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4519.971566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4519.976771] Lustre: Skipped 1 previous similar message [ 4520.038209] Lustre: lustre-MDT0001: new disk, initializing [ 4520.189683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4520.244937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4520.255479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4525.160234] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4536.147733] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4542.731575] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4549.221545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4549.667691] Lustre: lustre-OST0000: new disk, initializing [ 4549.671525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4549.679623] Lustre: 161486:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4549.800228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.355497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4555.379393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4555.554363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4557.223770] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4567.557902] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4571.878125] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4574.076296] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:56:09 (1788314169) [ 4832.240854] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.245032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4832.282527] Lustre: Skipped 2 previous similar messages [ 4832.284313] Lustre: Skipped 1 previous similar message [ 4834.678130] Lustre: server umount lustre-MDT0000 complete [ 4837.345634] LustreError: 160626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.382023] LustreError: 160626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4839.699961] LustreError: 160312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314436 with bad export cookie 1039275247738155226 [ 4839.710058] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4839.719481] LustreError: 160312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.464687] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.466124] LustreError: 160627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.493056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4842.536707] Lustre: Skipped 1 previous similar message [ 4844.513634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4849.633355] LustreError: 159523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.633598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4849.653495] LustreError: 159523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4854.240141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4854.488910] Lustre: server umount lustre-MDT0001 complete [ 4873.184327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4873.470080] Lustre: server umount lustre-OST0000 complete [ 4886.088921] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4898.534192] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4939.781115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.080458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4940.153729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4940.258504] Lustre: lustre-MDT0000: new disk, initializing [ 4940.321524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4940.337345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4946.172876] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4958.905745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.089792] Lustre: 166162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4959.183418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4959.187988] Lustre: Skipped 1 previous similar message [ 4959.271821] Lustre: lustre-MDT0001: new disk, initializing [ 4959.379724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.434170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4959.459115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4964.212111] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4973.434671] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4979.702229] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4988.642695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4989.094688] Lustre: lustre-OST0000: new disk, initializing [ 4989.100666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4989.108186] Lustre: 168131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4989.263851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.105917] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4998.738706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4998.762743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4998.875729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5009.151267] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5014.598938] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5014.847521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5019.423640] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5019.592162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5024.744681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.745033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5024.775901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5028.279805] Lustre: server umount lustre-OST0000 complete [ 5034.980138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5034.997254] Lustre: Skipped 1 previous similar message [ 5035.005039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5035.022569] Lustre: Skipped 2 previous similar messages [ 5038.829905] Lustre: server umount lustre-MDT0000 complete [ 5040.099050] LustreError: 166174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.136961] LustreError: 166174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5043.462423] LustreError: 166958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314640 with bad export cookie 1039275247738314049 [ 5043.469283] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.484094] LustreError: 166958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5044.201552] Lustre: server umount lustre-MDT0001 complete [ 5054.567832] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:04:09 (1788314649) [ 5074.480244] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5087.817060] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5130.496694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.719328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5130.762648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5130.832958] Lustre: lustre-MDT0000: new disk, initializing [ 5130.938809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.996131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5136.949938] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5149.725805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.881692] Lustre: 172889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5150.198639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5155.692831] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5166.519733] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5174.453044] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5183.614349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5184.011227] Lustre: lustre-OST0000: new disk, initializing [ 5184.014410] Lustre: Skipped 1 previous similar message [ 5184.023133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5184.029969] Lustre: Skipped 2 previous similar messages [ 5184.042190] Lustre: 174872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5191.254967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5191.271399] Lustre: Skipped 1 previous similar message [ 5191.283243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5191.504472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5192.373981] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5202.574245] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5212.132699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5212.140775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.143442] LustreError: Skipped 1 previous similar message [ 5212.144694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5212.172429] Lustre: Skipped 2 previous similar messages [ 5217.573800] Lustre: server umount lustre-OST0000 complete [ 5221.858284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5221.869546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.870255] Lustre: server umount lustre-MDT0000 complete [ 5232.285093] LustreError: 172882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314829 with bad export cookie 1039275247738314938 [ 5232.304699] LustreError: 172882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5232.307220] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5232.746119] Lustre: server umount lustre-MDT0001 complete [ 5242.975625] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:07:18 (1788314838) [ 5262.431647] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5273.466122] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5316.327027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.602644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5316.635566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5316.765533] Lustre: lustre-MDT0000: new disk, initializing [ 5316.862751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.873840] Lustre: Skipped 2 previous similar messages [ 5316.900888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5321.924573] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5333.993125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.269870] Lustre: 179196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5334.618368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5340.379709] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5350.802182] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5357.126945] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5364.393739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5364.768567] Lustre: 181171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5371.881519] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5372.507379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5372.574121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5382.221815] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5393.381152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5393.385699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.396523] LustreError: Skipped 1 previous similar message [ 5393.403456] Lustre: Skipped 1 previous similar message [ 5393.410261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5393.425302] Lustre: Skipped 9 previous similar messages [ 5404.128548] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5404.475214] Lustre: server umount lustre-OST0000 complete [ 5411.307600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5415.550234] Lustre: server umount lustre-MDT0000 complete [ 5418.981285] LustreError: 179203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.008119] LustreError: 179203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5420.014354] LustreError: 179187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315017 with bad export cookie 1039275247738316016 [ 5420.015567] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5420.036140] LustreError: 179187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5420.556942] Lustre: server umount lustre-MDT0001 complete [ 5433.372187] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5436.741211] Key type lgssc unregistered [ 5437.213825] LNet: 183033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5437.222766] LNetError: 183033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5437.247843] LNet: Removed LNI 192.168.204.145@tcp [ 5438.508505] Key type .llcrypt unregistered [ 5438.513111] Key type ._llcrypt unregistered [ 5465.179697] Key type ._llcrypt registered [ 5465.184770] Key type .llcrypt registered [ 5465.403056] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5466.543168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5466.593856] alg: No test for adler32 (adler32-zlib) [ 5467.686772] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5467.960524] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5469.696197] Key type lgssc registered [ 5470.974342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5480.039916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5480.082922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.605818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.177848] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5491.177364] LustreError: 184453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.210569] LustreError: 184453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5496.297922] LustreError: 184454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.349595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.764628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5504.138288] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5513.439574] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5519.515936] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5527.133857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.465867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.362706] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5543.882841] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5551.049625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5573.605271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.621536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.634367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5576.677152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5576.680766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5576.688760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.816935] Lustre: server umount lustre-OST0000 complete [ 5585.892751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5585.904211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.938605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.958413] Lustre: Skipped 1 previous similar message [ 5589.314926] Lustre: server umount lustre-MDT0000 complete [ 5592.034959] LustreError: 185183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.049226] LustreError: 185183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5593.651469] LustreError: 185191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315190 with bad export cookie 6305799622689173654 [ 5593.661635] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5593.664611] LustreError: 185191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5594.081659] Lustre: server umount lustre-MDT0001 complete [ 5606.624482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5609.961797] Key type lgssc unregistered [ 5610.354862] LNet: 188018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5610.361769] LNetError: 188018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5610.375042] LNet: Removed LNI 192.168.204.145@tcp [ 5611.450443] Key type .llcrypt unregistered [ 5611.454850] Key type ._llcrypt unregistered [ 5637.739687] Key type ._llcrypt registered [ 5637.744157] Key type .llcrypt registered [ 5637.929787] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5639.502258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5639.626796] alg: No test for adler32 (adler32-zlib) [ 5641.025088] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5641.395793] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5643.120283] Key type lgssc registered [ 5644.642380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5655.367833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5655.404856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.074558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5662.631357] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5667.307775] LustreError: 189435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.344244] LustreError: 189435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5672.410770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.418202] LustreError: 189436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.844455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5678.251718] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5686.924333] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5693.879191] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5702.308728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5702.676586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.763474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5709.688842] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5717.748397] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5724.343607] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 5727.201082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.216370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.227651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.235768] Lustre: Skipped 1 previous similar message [ 5727.248620] Lustre: Skipped 1 previous similar message [ 5729.454625] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 5729.914811] Lustre: server umount lustre-OST0000 complete [ 5738.007888] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff994991a4f800.ost_server_uuid 50 [ 5739.872459] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5753.254314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.266470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.271195] Lustre: Skipped 1 previous similar message [ 5758.433715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5758.439956] Lustre: Skipped 1 previous similar message [ 5759.598199] Lustre: server umount lustre-MDT0000 complete [ 5763.554313] LustreError: 189436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.580905] LustreError: 189436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5763.750156] LustreError: 189416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315360 with bad export cookie 1174291051236596970 [ 5763.756375] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.764557] LustreError: 189416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5764.264491] Lustre: server umount lustre-MDT0001 complete [ 5775.103890] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:16:10 (1788315370) [ 5784.007610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5784.632192] LustreError: 193354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5784.696682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5789.600902] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5790.178639] LustreError: 193355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.027441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.308154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5805.605919] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5814.673942] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.381391] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5829.563287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5830.187150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.196758] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5846.632732] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5854.490723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5854.754833] Lustre: lustre-OST0001: new disk, initializing [ 5854.763218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5854.765686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5854.782571] Lustre: 196197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5854.964180] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.218818] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5864.007850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5864.017346] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5864.122662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5872.254715] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5875.172315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5875.173099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5875.177933] LustreError: Skipped 1 previous similar message [ 5875.179166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5875.200877] Lustre: Skipped 2 previous similar messages [ 5881.376605] Lustre: server umount lustre-OST0000 complete [ 5883.028380] LustreError: 195177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.053970] LustreError: 195177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5888.771765] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5888.995542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5893.779837] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5894.053741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5900.270742] LustreError: 195177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.290952] LustreError: 195177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5910.498871] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5910.503858] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.517069] LustreError: Skipped 1 previous similar message [ 5910.519109] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5910.534198] Lustre: Skipped 1 previous similar message [ 5910.555193] Lustre: Skipped 5 previous similar messages [ 5914.366638] Lustre: server umount lustre-OST0001 complete [ 5920.740780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5920.758268] Lustre: Skipped 1 previous similar message [ 5924.547351] Lustre: server umount lustre-MDT0000 complete [ 5927.486584] LustreError: 196194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315524 with bad export cookie 1174291051236598041 [ 5927.491262] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5927.494939] LustreError: 196194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5927.895801] Lustre: server umount lustre-MDT0001 complete [ 5948.443352] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:19:04 (1788315544) [ 5954.065984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.294763] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5954.341718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5954.609659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.125956] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5967.757672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.946730] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5967.993327] Lustre: 199633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5973.531762] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5981.209551] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5988.010865] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5995.759656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.033662] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6002.861273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6003.661563] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6011.448730] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6018.351920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.406422] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6024.657914] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6031.854228] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6037.682233] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 6037.696799] Lustre: Skipped 2 previous similar messages [ 6037.995238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.011555] Lustre: Skipped 1 previous similar message [ 6040.559833] Lustre: server umount lustre-OST0000 complete [ 6042.773101] LustreError: 201486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.792091] LustreError: 201486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6058.473563] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6058.475386] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.485915] LustreError: Skipped 1 previous similar message [ 6058.517021] Lustre: Skipped 1 previous similar message [ 6062.646823] Lustre: server umount lustre-OST0001 complete [ 6072.747931] Lustre: server umount lustre-MDT0000 complete [ 6076.372043] LustreError: 199626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315673 with bad export cookie 1174291051236599448 [ 6076.375774] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6076.383468] LustreError: 199626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6076.740847] Lustre: server umount lustre-MDT0001 complete [ 6100.533801] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:21:36 (1788315696) [ 6118.223546] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 6128.931936] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6170.078843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.440454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6170.487397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6170.532289] Lustre: lustre-MDT0000: new disk, initializing [ 6170.616267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.628598] Lustre: Skipped 3 previous similar messages [ 6170.650406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6176.348821] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6186.736758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.820475] Lustre: 207736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6186.913057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6186.921298] Lustre: Skipped 1 previous similar message [ 6187.011366] Lustre: lustre-MDT0001: new disk, initializing [ 6187.132987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6187.152657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6191.181643] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6201.717705] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6209.343799] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.034692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.413367] Lustre: lustre-OST0000: new disk, initializing [ 6217.416395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6217.420710] Lustre: 209703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6223.336253] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6227.586448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6227.603372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6227.699192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6231.883541] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6236.873927] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6237.202772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6242.158561] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6242.481476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6248.423649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6248.425237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.435258] LustreError: Skipped 1 previous similar message [ 6248.440885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6248.460181] Lustre: Skipped 3 previous similar messages [ 6248.485787] Lustre: Skipped 7 previous similar messages [ 6250.950490] Lustre: server umount lustre-OST0000 complete [ 6258.672996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6261.164330] Lustre: server umount lustre-MDT0000 complete [ 6263.777680] LustreError: 208958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.798731] LustreError: 208958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6265.208083] LustreError: 207728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315862 with bad export cookie 1174291051236601009 [ 6265.217579] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6265.219511] LustreError: 207728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6265.537527] Lustre: server umount lustre-MDT0001 complete [ 6274.100948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.225328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6288.529169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.028109] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6299.811634] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6305.058220] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6311.187526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6311.545144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6311.553393] Lustre: Skipped 4 previous similar messages [ 6316.910134] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6323.855337] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6329.117452] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6329.298788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6334.757273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6334.998909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6339.560426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6339.570917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.581221] Lustre: Skipped 2 previous similar messages [ 6351.328821] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6351.485390] Lustre: server umount lustre-OST0000 complete [ 6357.996931] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6362.326475] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6391.742243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.173116] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6406.514355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.557972] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.566354] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6408.056114] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6410.116419] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6416.867556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6416.878280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.882412] LustreError: Skipped 2 previous similar messages [ 6416.892491] Lustre: Skipped 10 previous similar messages [ 6420.327447] Lustre: server umount lustre-OST0000 complete [ 6430.691701] Lustre: server umount lustre-MDT0000 complete [ 6432.234374] LustreError: 211919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.261901] LustreError: 211919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6435.762406] LustreError: 211899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316032 with bad export cookie 1174291051236601891 [ 6435.771629] LustreError: 211899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6435.781389] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6436.187690] Lustre: server umount lustre-MDT0001 complete [ 6448.103701] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:27:23 (1788316043) [ 6455.581420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.821064] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6471.010568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.727930] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6483.794483] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6489.633187] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6496.436844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6502.889655] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6509.810737] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6514.665969] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6514.865358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6521.055420] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6521.387348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6529.234150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6529.471967] Lustre: lustre-OST0001: new disk, initializing [ 6529.475067] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6529.480038] Lustre: 221263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6537.678855] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6537.901589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6537.915612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6538.069915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6546.505303] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6550.727359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6550.910909] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6555.224541] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6555.509626] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6558.691987] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6558.694719] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.711538] LustreError: Skipped 1 previous similar message [ 6558.738132] Lustre: Skipped 6 previous similar messages [ 6563.379531] Lustre: server umount lustre-OST0001 complete [ 6569.796870] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6570.049181] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6573.847413] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6574.028244] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6605.034324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.339569] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6605.352746] Lustre: Skipped 5 previous similar messages [ 6610.413655] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6610.434126] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6610.439040] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6610.466506] Lustre: Skipped 2 previous similar messages [ 6613.410876] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6621.629609] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6623.340469] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6631.512504] Lustre: server umount lustre-OST0001 complete [ 6655.896415] Lustre: server umount lustre-OST0000 complete [ 6666.442852] Lustre: server umount lustre-MDT0000 complete [ 6670.817949] LustreError: 217872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316268 with bad export cookie 1174291051236602857 [ 6670.821861] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6670.836761] LustreError: 217872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6685.152124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6685.372541] Lustre: server umount lustre-MDT0001 complete [ 6708.728767] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:31:44 (1788316304) [ 6715.575757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.842333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6715.931166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6721.062237] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6729.551509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.643168] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6729.703894] Lustre: 226509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6736.055695] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6745.909314] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6752.911988] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6759.899880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6760.139832] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6765.292892] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6771.516619] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6780.220467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6780.309024] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6786.533969] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6794.293604] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6795.829457] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6800.556781] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6800.777994] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6804.948067] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6805.194844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6810.490742] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99498837a800.ost_server_uuid 50 [ 6812.105197] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99498837a800.ost_server_uuid in IDLE state after 0 sec [ 6813.284912] Lustre: Permanently deactivating lustre-OST0001 [ 6813.299895] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6816.983142] Lustre: Permanently reactivating lustre-OST0001 [ 6823.905295] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6823.906656] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.912855] LustreError: Skipped 5 previous similar messages [ 6823.915993] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6823.932984] Lustre: Skipped 7 previous similar messages [ 6823.945270] Lustre: Skipped 14 previous similar messages [ 6827.275091] Lustre: server umount lustre-OST0001 complete [ 6829.036159] LustreError: 228354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.061658] LustreError: 228354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 6837.504791] Lustre: server umount lustre-OST0000 complete [ 6847.758853] Lustre: server umount lustre-MDT0000 complete [ 6851.644500] LustreError: 226545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316448 with bad export cookie 1174291051236604138 [ 6851.649160] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6851.656507] LustreError: 226545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6851.923912] Lustre: server umount lustre-MDT0001 complete [ 6875.824301] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:34:31 (1788316471) [ 6883.719382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.950583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6883.986742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6883.999212] Lustre: Skipped 5 previous similar messages [ 6889.083680] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6897.861816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.985210] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6898.045253] Lustre: 233200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6902.825663] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6908.655113] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6912.629287] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6919.188674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.493200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6919.541749] Lustre: Permanently deactivating lustre-OST0000 [ 6919.558906] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6926.418670] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6928.385831] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6932.953938] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6939.019892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6939.111976] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6945.729823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6947.998035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6953.098210] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.600501] Lustre: Permanently reactivating lustre-OST0000 [ 6954.614106] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6954.618314] Lustre: Skipped 2 previous similar messages [ 6960.713095] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6960.725384] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000003044b6b8, cur 1788316557 deadline 1788316555 last 1788316525 [ 6960.794629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6985.185393] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6985.336081] Lustre: server umount lustre-OST0001 complete [ 6995.196980] Lustre: server umount lustre-OST0000 complete [ 7004.929214] Lustre: server umount lustre-MDT0000 complete [ 7009.340342] LustreError: 233193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316606 with bad export cookie 1174291051236605818 [ 7009.344707] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7009.355782] LustreError: 233193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7009.822869] Lustre: server umount lustre-MDT0001 complete [ 7023.899986] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7027.569798] Key type lgssc unregistered [ 7028.005783] LNet: 238084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7028.011857] LNetError: 238084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7028.041082] LNet: Removed LNI 192.168.204.145@tcp [ 7028.984676] Key type .llcrypt unregistered [ 7028.986734] Key type ._llcrypt unregistered [ 7052.323355] Key type ._llcrypt registered [ 7052.326360] Key type .llcrypt registered [ 7052.462677] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7053.120268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7053.140154] alg: No test for adler32 (adler32-zlib) [ 7054.132630] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7054.306326] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7055.960236] Key type lgssc registered [ 7057.190736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7068.976100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7069.004119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.650912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.324364] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7079.910975] LustreError: 239555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.945879] LustreError: 239555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7085.030042] LustreError: 239556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.302341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.482475] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7085.567970] Lustre: 239544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7085.611568] Lustre: Permanently deactivating lustre-MDT0001 [ 7085.661247] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7086.089678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7091.804487] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7094.314248] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7099.891242] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7106.501641] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.099347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.491123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7115.571771] LustreError: 241398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7115.598967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7120.866968] LustreError: 241399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.881771] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7125.997853] LustreError: 241400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.664343] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7131.153915] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7136.243804] LustreError: 241399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.278958] LustreError: 241399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7140.159758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7140.396972] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7145.508246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7145.512951] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7147.021994] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7153.863461] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7156.590637] Lustre: Permanently reactivating lustre-MDT0001 [ 7156.607790] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7156.621916] Lustre: Skipped 2 previous similar messages [ 7159.987672] Lustre: lustre-MDT0001: Client 54c975ee-92cb-4d63-b98f-f1f398868f88 (at 192.168.204.45@tcp) reconnecting [ 7162.542270] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.567160] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7162.576039] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7162.591159] LustreError: 243211:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7162.598331] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7162.599320] LustreError: 243211:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7162.599454] LustreError: 243211:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -108 [ 7162.617954] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7162.617954] [ 7162.667697] LustreError: 239548:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7167.436589] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7169.609398] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7177.161424] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7177.433667] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7183.686391] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7183.982611] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7186.999912] Lustre: Permanently deactivating lustre-MDT0001 [ 7187.017466] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7187.024599] Lustre: Skipped 2 previous similar messages [ 7193.693664] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7202.879052] Lustre: server umount lustre-MDT0000 complete [ 7203.813956] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.820401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7203.832382] Lustre: Skipped 2 previous similar messages [ 7203.838589] LustreError: 239551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.868905] LustreError: 239551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7206.495641] LustreError: 239536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316803 with bad export cookie 14641004467037440601 [ 7206.500575] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.504684] LustreError: 239536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7206.790829] Lustre: server umount lustre-MDT0001 complete [ 7212.995954] Lustre: server umount lustre-OST0000 complete [ 7226.349769] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:40:21 (1788316821) [ 7238.233320] Lustre: server umount lustre-OST0001 complete [ 7244.759493] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7255.184709] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7296.469705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.792154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7296.799418] Lustre: Skipped 2 previous similar messages [ 7296.819789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7296.922811] Lustre: lustre-MDT0000: new disk, initializing [ 7297.046431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.084653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7302.497835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7314.551601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7314.699960] Lustre: 247818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7314.787096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7314.795170] Lustre: Skipped 1 previous similar message [ 7314.939276] Lustre: lustre-MDT0001: new disk, initializing [ 7315.149864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7315.210773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7315.227929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7320.460593] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7329.822588] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7335.294613] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7341.723912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7342.086221] Lustre: lustre-OST0000: new disk, initializing [ 7342.091128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7342.100400] Lustre: 249785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7342.170153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7347.931695] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7351.893751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7351.907185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7352.028784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7356.492453] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7358.119410] LustreError: 247824:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7366.740374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.942451] Lustre: lustre-OST0001: new disk, initializing [ 7366.946893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7366.956738] Lustre: 250899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7368.145736] LustreError: 247824:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7368.188279] LustreError: 247825:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7374.454611] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7376.492369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7376.501700] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7376.657717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7378.190769] LustreError: 247825:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7384.946389] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7386.560375] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7392.229771] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7392.230266] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.236827] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7392.249577] LustreError: Skipped 1 previous similar message [ 7392.274111] Lustre: Skipped 2 previous similar messages [ 7394.634866] Lustre: server umount lustre-OST0001 complete [ 7397.347725] LustreError: 249769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.380620] LustreError: 249769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7412.729250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.741762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7412.752871] Lustre: Skipped 2 previous similar messages [ 7417.647836] Lustre: server umount lustre-OST0000 complete [ 7422.434164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7422.441639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.455343] Lustre: Skipped 1 previous similar message [ 7422.459828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.154888] Lustre: server umount lustre-MDT0000 complete [ 7432.758867] LustreError: 247811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317029 with bad export cookie 14641004467037443415 [ 7432.760094] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7432.763971] LustreError: 247811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7439.258541] Lustre: server umount lustre-MDT0001 complete [ 7451.520957] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7455.267435] Key type lgssc unregistered [ 7455.746610] LNet: 252941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7455.758937] LNetError: 252941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7455.789442] LNet: Removed LNI 192.168.204.145@tcp [ 7457.059588] Key type .llcrypt unregistered [ 7457.061671] Key type ._llcrypt unregistered [ 7501.124435] Key type ._llcrypt registered [ 7501.131706] Key type .llcrypt registered [ 7501.425559] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7503.576603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7503.665330] alg: No test for adler32 (adler32-zlib) [ 7504.770744] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7505.114723] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7506.872472] Key type lgssc registered [ 7508.056486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7517.003687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7517.067883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.579151] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7518.621992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7518.839244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.928929] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7534.357701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7534.499282] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7534.556327] Lustre: 255152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7534.875265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7539.608658] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7547.646503] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7553.994194] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7560.815954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.966136] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7561.149407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.022243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7568.671616] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7577.328381] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7633.381033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7633.394657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.411635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7633.903099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7633.911837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.930161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7636.471873] Lustre: server umount lustre-OST0000 complete [ 7641.221645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7654.063023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7654.523418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7660.004914] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7660.030518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7660.035401] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7660.059093] Lustre: Skipped 1 previous similar message [ 7660.454593] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7667.388761] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.088187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7706.090938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.093972] LustreError: Skipped 1 previous similar message [ 7706.096012] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7706.133088] Lustre: Skipped 1 previous similar message [ 7711.162863] Lustre: server umount lustre-OST0000 complete [ 7716.329549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.339413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.359260] Lustre: Skipped 1 previous similar message [ 7716.376912] Lustre: Skipped 2 previous similar messages [ 7721.446041] LustreError: 255163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.475118] LustreError: 255163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7721.917843] Lustre: server umount lustre-MDT0000 complete [ 7726.164575] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317323 with bad export cookie 18335635301203629772 [ 7726.172082] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7726.183425] LustreError: 255144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7726.458438] Lustre: server umount lustre-MDT0001 complete [ 7738.820469] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7741.101729] Key type lgssc unregistered [ 7741.321615] LNet: 260580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7741.331230] LNetError: 260580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7741.354803] LNet: Removed LNI 192.168.204.145@tcp [ 7742.078094] Key type .llcrypt unregistered [ 7742.085475] Key type ._llcrypt unregistered [ 7766.289322] Key type ._llcrypt registered [ 7766.291107] Key type .llcrypt registered [ 7766.396698] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7767.850665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7767.912677] alg: No test for adler32 (adler32-zlib) [ 7768.929214] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7769.231110] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7770.889291] Key type lgssc registered [ 7772.260812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7781.799625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7781.826114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.505400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.550322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7793.643837] LustreError: 261998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7793.667877] LustreError: 261998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7797.162699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.537255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7801.151356] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7808.172600] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7814.098200] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7822.814663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7823.274103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.413676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7830.916508] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7838.376211] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7863.268864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7863.271208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.287884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7863.318898] Lustre: Skipped 1 previous similar message [ 7868.394509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7868.408398] Lustre: Skipped 2 previous similar messages [ 7869.369985] Lustre: server umount lustre-OST0000 complete [ 7873.509405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.515729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7873.522659] Lustre: Skipped 1 previous similar message [ 7873.537153] Lustre: Skipped 1 previous similar message [ 7878.628622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.645921] Lustre: Skipped 1 previous similar message [ 7879.703368] Lustre: server umount lustre-MDT0000 complete [ 7883.745962] LustreError: 261999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.759671] LustreError: 261999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7883.914131] LustreError: 261977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317481 with bad export cookie 12592763153145823167 [ 7883.917357] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.930859] LustreError: 261977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7884.314305] Lustre: server umount lustre-MDT0001 complete [ 7895.914397] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7898.926735] Key type lgssc unregistered [ 7899.292731] LNet: 266307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7899.309187] LNetError: 266307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7899.326801] LNet: Removed LNI 192.168.204.145@tcp [ 7900.405215] Key type .llcrypt unregistered [ 7900.408142] Key type ._llcrypt unregistered [ 7916.743100] Key type ._llcrypt registered [ 7916.747217] Key type .llcrypt registered [ 7916.878543] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7917.511769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7917.560592] alg: No test for adler32 (adler32-zlib) [ 7918.746928] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7919.119044] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7920.784939] Key type lgssc registered [ 7922.244815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7930.720619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7930.754919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.109064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.828474] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7942.637368] LustreError: 267548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7942.660038] LustreError: 267548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7947.748608] LustreError: 267549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.138697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.795458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7955.980470] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7965.346478] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7969.760243] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7976.512983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7977.065480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.096399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7984.764950] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7991.659641] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8009.193530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.199605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8009.202022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.214418] Lustre: Skipped 1 previous similar message [ 8009.240747] LustreError: Skipped 1 previous similar message [ 8013.547856] Lustre: server umount lustre-OST0000 complete [ 8019.434869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.449274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8019.462161] Lustre: Skipped 1 previous similar message [ 8019.476679] Lustre: Skipped 2 previous similar messages [ 8024.566468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8024.575816] Lustre: Skipped 1 previous similar message [ 8025.124829] Lustre: server umount lustre-MDT0000 complete [ 8029.307188] LustreError: 269391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317626 with bad export cookie 7214062895386583585 [ 8029.324196] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8029.336442] LustreError: 269391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8029.642549] Lustre: server umount lustre-MDT0001 complete [ 8040.300113] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8043.618818] Key type lgssc unregistered [ 8043.995636] LNet: 271232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8044.007448] LNetError: 271232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8045.031828] LNet: Removed LNI 192.168.204.145@tcp [ 8046.137234] Key type .llcrypt unregistered [ 8046.144096] Key type ._llcrypt unregistered [ 8072.857094] Key type ._llcrypt registered [ 8072.859178] Key type .llcrypt registered [ 8072.962322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8073.982355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8074.098041] alg: No test for adler32 (adler32-zlib) [ 8075.219852] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 8075.442905] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8077.161768] Key type lgssc registered [ 8078.384847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8086.088380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8086.104799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.534833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8092.845384] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8097.763585] LustreError: 272647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.774634] LustreError: 272647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8101.609444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.996814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8107.156568] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8115.519826] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8120.691454] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8129.209298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8129.645806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.744275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8137.120208] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8147.749042] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8173.025598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8173.033694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.043527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8173.066525] Lustre: Skipped 1 previous similar message [ 8173.086218] Lustre: Skipped 1 previous similar message [ 8178.147919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8178.168436] Lustre: Skipped 1 previous similar message [ 8179.072839] Lustre: server umount lustre-OST0000 complete [ 8183.778680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.786542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.810938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.392041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8188.400488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.871982] Lustre: server umount lustre-MDT0000 complete [ 8193.505796] LustreError: 272648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8193.525676] LustreError: 272648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8193.809793] LustreError: 272629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317791 with bad export cookie 16265411601158940681 [ 8193.827297] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.828299] LustreError: 272629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8194.355628] Lustre: server umount lustre-MDT0001 complete [ 8207.212542] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8210.307687] Key type lgssc unregistered [ 8210.763675] LNet: 276960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8210.775530] LNetError: 276960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8211.819594] LNet: Removed LNI 192.168.204.145@tcp [ 8212.933248] Key type .llcrypt unregistered [ 8212.938467] Key type ._llcrypt unregistered [ 8233.346613] Key type ._llcrypt registered [ 8233.352341] Key type .llcrypt registered [ 8233.476492] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8235.234976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8235.295361] alg: No test for adler32 (adler32-zlib) [ 8236.342672] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 8236.650354] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8238.400247] Key type lgssc registered [ 8239.555837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8247.913590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8247.951548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.486469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.666464] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8259.051509] LustreError: 278201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.067904] LustreError: 278201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8264.162435] LustreError: 278203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8264.522111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8265.024619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8271.002188] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8280.632953] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8287.296418] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8296.107215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8296.506895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8304.185911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8312.192972] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8320.054921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8334.618735] Lustre: server umount lustre-OST0000 complete [ 8340.450169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8340.454106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.470708] Lustre: Skipped 1 previous similar message [ 8340.489561] Lustre: Skipped 1 previous similar message [ 8345.574084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.584351] Lustre: Skipped 1 previous similar message [ 8348.783843] Lustre: server umount lustre-MDT0000 complete [ 8350.689795] LustreError: 278184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.715203] LustreError: 278184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8351.712118] Lustre: 277496:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788317932/real 1788317932] req@00000000890f4fa0 x1875187160087808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788317948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8351.729761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8353.032821] LustreError: 278167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788317950 with bad export cookie 1104521895255045619 [ 8353.045336] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.047398] LustreError: 278167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8353.464860] Lustre: server umount lustre-MDT0001 complete [ 8366.518911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8370.333686] Key type lgssc unregistered [ 8370.967566] LNet: 281926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8370.978633] LNetError: 281926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8371.018939] LNet: Removed LNI 192.168.204.145@tcp [ 8372.257313] Key type .llcrypt unregistered [ 8372.259866] Key type ._llcrypt unregistered [ 8386.409782] Key type ._llcrypt registered [ 8386.413049] Key type .llcrypt registered [ 8386.603973] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8404.059506] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8422.505414] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8424.052103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8424.158186] alg: No test for adler32 (adler32-zlib) [ 8425.408863] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 8425.691272] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8427.392342] Key type lgssc registered [ 8428.874593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8470.796593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8470.857876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.246852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8472.282733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8472.368641] Lustre: lustre-MDT0000: new disk, initializing [ 8472.520809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8472.565780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8477.924704] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8490.815580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.928180] Lustre: 285492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8490.998705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8491.003700] Lustre: Skipped 1 previous similar message [ 8491.089607] Lustre: lustre-MDT0001: new disk, initializing [ 8491.250729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.305762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8491.312890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8496.188984] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8507.048192] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8514.995356] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8522.814621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8523.272858] Lustre: lustre-OST0000: new disk, initializing [ 8523.276881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8523.288359] Lustre: 287460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8523.405579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8530.485740] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8533.076499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8533.083490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8533.179218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8540.824604] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8545.490853] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8545.726379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8550.289657] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8550.550384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8553.957031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.959101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8553.981808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.998088] Lustre: Skipped 1 previous similar message [ 8558.965773] Lustre: server umount lustre-OST0000 complete [ 8564.197442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.211637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8564.214610] Lustre: Skipped 2 previous similar messages [ 8564.237707] Lustre: Skipped 1 previous similar message [ 8569.315614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8569.326481] Lustre: Skipped 1 previous similar message [ 8569.516150] Lustre: server umount lustre-MDT0000 complete [ 8573.446882] LustreError: 285484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318170 with bad export cookie 2944166735118760852 [ 8573.453474] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.462980] LustreError: 285484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8574.134379] Lustre: server umount lustre-MDT0001 complete [ 8583.673716] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:02:59 (1788318179) [ 8590.228626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.790076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8595.434417] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8595.944570] LustreError: 289872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.960059] LustreError: 289872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8600.039326] LustreError: 289871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8604.502987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.830417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.201038] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8618.126728] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8624.474680] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8631.349188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8631.773483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8638.216695] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8646.101959] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8659.447031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8659.449164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.458552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8659.478844] Lustre: Skipped 1 previous similar message [ 8664.555389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8664.570254] Lustre: Skipped 2 previous similar messages [ 8665.388839] Lustre: server umount lustre-OST0000 complete [ 8669.665485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.674992] Lustre: Skipped 1 previous similar message [ 8674.788564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8674.803688] Lustre: Skipped 3 previous similar messages [ 8675.217627] Lustre: server umount lustre-MDT0000 complete [ 8678.870124] LustreError: 289852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318276 with bad export cookie 2944166735118761734 [ 8678.878192] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8678.888570] LustreError: 289852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8679.247986] Lustre: server umount lustre-MDT0001 complete [ 8690.775554] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8693.662209] Key type lgssc unregistered [ 8694.081753] LNet: 293432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8694.089062] LNetError: 293432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8694.135341] LNet: Removed LNI 192.168.204.145@tcp [ 8695.130583] Key type .llcrypt unregistered [ 8695.132739] Key type ._llcrypt unregistered [ 8723.152460] Key type ._llcrypt registered [ 8723.154988] Key type .llcrypt registered [ 8723.266650] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8740.348181] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8741.716866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8741.798092] alg: No test for adler32 (adler32-zlib) [ 8743.202140] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 8743.569915] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8745.312636] Key type lgssc registered [ 8746.879533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8795.774853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8795.802198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.023915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8797.046979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8797.124444] Lustre: lustre-MDT0000: new disk, initializing [ 8797.200640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8797.214092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8801.346823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8812.292511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.423275] Lustre: 297261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8812.489578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8812.493225] Lustre: Skipped 1 previous similar message [ 8812.594403] Lustre: lustre-MDT0001: new disk, initializing [ 8812.735121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8812.768689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8812.789173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8817.598337] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8828.257581] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8835.084644] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8842.898836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8843.275739] Lustre: lustre-OST2710: new disk, initializing [ 8843.287388] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8843.297919] Lustre: 299229:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8843.433922] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8849.754683] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8850.529157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8850.544817] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8850.731787] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8860.488602] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8868.268935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8868.550450] Lustre: lustre-OST03e8: new disk, initializing [ 8868.556996] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8868.568825] Lustre: 300296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.700266] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8875.112038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8875.126475] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8875.279555] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8875.455260] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8883.209443] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8891.505845] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8891.829125] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8897.260592] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8897.456407] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8902.664150] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8902.932263] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8907.498746] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8907.742453] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8914.187934] Lustre: server umount lustre-MDT0000 complete [ 8915.428432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8915.437443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.456864] LustreError: 297271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8915.479983] LustreError: 297271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8916.455328] LustreError: 298342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8916.459402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.484498] LustreError: 298342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8916.505304] Lustre: Skipped 1 previous similar message [ 8918.084993] LustreError: 297253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318515 with bad export cookie 6334081669909147716 [ 8918.096812] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.101670] LustreError: 297253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8918.430843] Lustre: server umount lustre-MDT0001 complete [ 8932.307502] Lustre: server umount lustre-OST2710 complete [ 8946.514694] Lustre: server umount lustre-OST03e8 complete [ 8965.416136] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 8975.483196] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9012.238644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.489695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9012.526648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9012.598693] Lustre: lustre-MDT0000: new disk, initializing [ 9012.712041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9012.734097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9017.684777] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9029.285652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.458630] Lustre: 305545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9029.551605] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9029.556120] Lustre: Skipped 1 previous similar message [ 9029.664594] Lustre: lustre-MDT0001: new disk, initializing [ 9029.860964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9029.928246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9029.949181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9036.290237] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9046.506923] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9052.410320] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9057.897906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9058.188183] Lustre: lustre-OST0000: new disk, initializing [ 9058.196126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9058.208695] Lustre: 307515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9058.261558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.934659] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9067.692705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9067.714892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9067.887718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9077.255774] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9082.195546] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9082.516329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9087.760655] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9087.969951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9093.601294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9093.605249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.616644] LustreError: Skipped 1 previous similar message [ 9093.618222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9093.637474] Lustre: Skipped 2 previous similar messages [ 9093.692351] Lustre: Skipped 1 previous similar message [ 9096.501821] Lustre: server umount lustre-OST0000 complete [ 9101.802048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9101.813725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9101.826696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9103.842255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9103.849234] Lustre: Skipped 1 previous similar message [ 9107.203449] Lustre: server umount lustre-MDT0000 complete [ 9108.965659] LustreError: 306628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9108.981557] LustreError: 306628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9111.437764] LustreError: 306351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318708 with bad export cookie 6334081669909149396 [ 9111.440146] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9111.449256] LustreError: 306351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9111.780201] Lustre: server umount lustre-MDT0001 complete [ 9121.016991] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:11:56 (1788318716) [ 9122.599460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9124.332881] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:12:00 (1788318720) [ 9128.021303] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9156.479496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9156.767395] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9156.830793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9157.052262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9162.023835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9171.869553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.011791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9172.070929] Lustre: 311371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9172.235925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9177.860650] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9200.638227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9200.853079] Lustre: Denying initial registration attempt from nid 192.168.204.145@tcp, specified as failover [ 9200.859500] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9200.868311] LustreError: 312826:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9200.876348] LustreError: 312826:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9200.883713] LustreError: 312826:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9200.952123] Lustre: server umount lustre-OST0000 complete [ 9200.956457] LustreError: 312826:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9213.409534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.410792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.428984] Lustre: Skipped 1 previous similar message [ 9213.434072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.458827] Lustre: Skipped 1 previous similar message [ 9214.898783] Lustre: server umount lustre-MDT0000 complete [ 9218.548162] LustreError: 311379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9218.582231] LustreError: 311379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9219.073953] LustreError: 311364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788318816 with bad export cookie 6334081669909150278 [ 9219.079185] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9219.083939] LustreError: 311364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9219.423213] Lustre: server umount lustre-MDT0001 complete [ 9232.424780] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9236.002526] Key type lgssc unregistered [ 9236.300160] LNet: 313950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9236.311055] LNetError: 313950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9236.328265] LNet: Removed LNI 192.168.204.145@tcp [ 9237.382780] Key type .llcrypt unregistered [ 9237.385231] Key type ._llcrypt unregistered [ 9250.409605] Key type ._llcrypt registered [ 9250.412403] Key type .llcrypt registered [ 9250.559725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9251.517845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9251.625440] alg: No test for adler32 (adler32-zlib) [ 9252.784077] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 9253.095833] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9254.832703] Key type lgssc registered [ 9256.284293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9305.582222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9318.968087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9319.013169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.563501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9320.597543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9320.805914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9326.050800] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9335.562405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.661273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9335.702900] Lustre: 316716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9335.960027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9340.977388] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9348.748951] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9354.962862] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9375.288205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9375.507568] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9375.770263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9383.381904] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9391.430164] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9394.663066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.664354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9394.703804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9399.784251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9399.797628] Lustre: Skipped 2 previous similar messages [ 9400.118336] Lustre: server umount lustre-OST0000 complete [ 9404.916044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9404.926091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9404.930299] Lustre: Skipped 2 previous similar messages [ 9404.945252] Lustre: Skipped 1 previous similar message [ 9410.017821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9410.032995] Lustre: Skipped 1 previous similar message [ 9415.138463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9415.147821] Lustre: Skipped 1 previous similar message [ 9418.208671] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9418.439081] Lustre: server umount lustre-MDT0000 complete [ 9420.258799] LustreError: 316729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9420.271159] LustreError: 316729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9422.111371] LustreError: 316709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319019 with bad export cookie 17782223499830850875 [ 9422.121863] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9422.131261] LustreError: 316709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9422.492334] Lustre: server umount lustre-MDT0001 complete [ 9436.188929] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9439.055333] Key type lgssc unregistered [ 9439.346956] LNet: 320297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9439.355589] LNetError: 320297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9439.379312] LNet: Removed LNI 192.168.204.145@tcp [ 9440.391415] Key type .llcrypt unregistered [ 9440.398322] Key type ._llcrypt unregistered [ 9464.054974] Key type ._llcrypt registered [ 9464.058890] Key type .llcrypt registered [ 9464.189333] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9465.323144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9465.389290] alg: No test for adler32 (adler32-zlib) [ 9466.875393] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 9467.218264] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9469.000213] Key type lgssc registered [ 9470.129816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9478.813951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9478.853113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.408645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9484.998222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9489.891219] LustreError: 321717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9489.904751] LustreError: 321717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9494.462280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.019123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9500.300824] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9506.822128] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9511.363242] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9519.020518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9519.397368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.158139] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9531.688780] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9538.728319] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 9541.097726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.099394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9541.100647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9541.108455] Lustre: Skipped 2 previous similar messages [ 9543.835698] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 9543.845925] Lustre: Skipped 2 previous similar messages [ 9544.563637] Lustre: server umount lustre-MDT0000 complete [ 9546.209796] LustreError: 321717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9546.232317] LustreError: 321717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9547.816839] LustreError: 321697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319145 with bad export cookie 16653636207169790782 [ 9547.829651] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.841609] LustreError: 321697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9548.946129] LustreError: 321712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9548.964447] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 9551.331739] LustreError: 323151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.332047] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9554.062041] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 9554.071797] Lustre: Skipped 2 previous similar messages [ 9554.178352] Lustre: server umount lustre-MDT0001 complete [ 9560.584940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9569.167865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.793772] LustreError: 321026:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee6a8242 x1875188450892544/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9574.188513] LustreError: 323550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.230399] LustreError: 323550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9574.282425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9574.329505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9579.235461] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9582.498907] LustreError: 325312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9582.504325] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9582.505932] LustreError: 325312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9587.704662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.091500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9592.235518] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9593.337889] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9599.433330] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9600.845799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9606.407359] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9607.945696] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9613.802709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9613.803977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9613.805619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9613.838539] Lustre: Skipped 1 previous similar message [ 9618.802217] Lustre: server umount lustre-OST0000 complete [ 9624.033534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9624.040754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.061123] Lustre: Skipped 1 previous similar message [ 9634.273363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9634.280189] Lustre: Skipped 6 previous similar messages [ 9636.320195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9636.462058] Lustre: server umount lustre-MDT0000 complete [ 9639.399353] LustreError: 325310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9639.420701] LustreError: 325310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9639.759521] LustreError: 321699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319236 with bad export cookie 16653636207169798461 [ 9639.760046] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9639.772099] LustreError: 321699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9640.045686] Lustre: server umount lustre-MDT0001 complete [ 9649.638528] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9652.664441] Key type lgssc unregistered [ 9652.976580] LNet: 327988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9652.986487] LNetError: 327988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9653.000517] LNet: Removed LNI 192.168.204.145@tcp [ 9653.701048] Key type .llcrypt unregistered [ 9653.704923] Key type ._llcrypt unregistered [ 9680.433475] Key type ._llcrypt registered [ 9680.437834] Key type .llcrypt registered [ 9680.572482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9681.123690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9681.143059] alg: No test for adler32 (adler32-zlib) [ 9682.147789] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 9682.375234] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9684.056304] Key type lgssc registered [ 9685.162399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9694.840212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9694.849705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.292786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9701.530303] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9706.467044] LustreError: 329422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.483683] LustreError: 329422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9710.384492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.917184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9716.361339] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9724.165987] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9730.907374] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9735.524791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9735.555367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9740.791923] LustreError: 330387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.831239] LustreError: 330387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9741.077853] Lustre: server umount lustre-MDT0000 complete [ 9745.379948] LustreError: 329401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319342 with bad export cookie 1142879756741559828 [ 9745.383936] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9745.399899] LustreError: 329401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9745.740936] Lustre: server umount lustre-MDT0001 complete [ 9759.447459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.782751] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9759.846514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9760.129297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.795490] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9773.293769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.460945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9773.501873] Lustre: 332009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9773.903942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9778.884716] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9786.830709] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9792.852824] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9802.040586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9802.236569] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9802.252475] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9802.263185] LustreError: 333894:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9802.270791] LustreError: 333894:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9802.276772] LustreError: 333894:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9802.373347] Lustre: server umount lustre-OST0000 complete [ 9802.375381] LustreError: 333894:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9813.563958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9813.914885] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9814.170258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9820.802334] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9823.980095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9829.119726] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9838.033485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9838.174448] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9838.305613] Lustre: lustre-OST0001: new disk, initializing [ 9838.312826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9838.338640] Lustre: 335437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9838.520987] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.578531] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9846.369799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9846.386205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9846.536771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9852.386896] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9856.994544] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9857.002126] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.004107] LustreError: Skipped 1 previous similar message [ 9857.016639] Lustre: Skipped 1 previous similar message [ 9857.018798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9857.032694] Lustre: Skipped 2 previous similar messages [ 9861.495771] Lustre: server umount lustre-OST0001 complete [ 9862.117702] LustreError: 334855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.158525] LustreError: 334855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9867.281760] LustreError: 334363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9867.286186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.293644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9867.297805] LustreError: 334363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9867.321273] Lustre: Skipped 2 previous similar messages [ 9872.407535] Lustre: server umount lustre-OST0000 complete [ 9876.449633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9876.456367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9876.480953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9876.489231] Lustre: Skipped 1 previous similar message [ 9882.474205] Lustre: server umount lustre-MDT0000 complete [ 9882.602392] LustreError: 332022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9882.614887] LustreError: 332022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9886.417823] LustreError: 332001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319483 with bad export cookie 1142879756741560381 [ 9886.426184] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9886.430941] LustreError: 332001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9886.904987] Lustre: server umount lustre-MDT0001 complete [ 9897.594895] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9900.558157] Key type lgssc unregistered [ 9900.915098] LNet: 337400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9900.926945] LNetError: 337400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9901.992944] LNet: Removed LNI 192.168.204.145@tcp [ 9903.015716] Key type .llcrypt unregistered [ 9903.019382] Key type ._llcrypt unregistered [ 9975.415948] Key type ._llcrypt registered [ 9975.418458] Key type .llcrypt registered [ 9975.507085] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9990.917232] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9992.134603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9992.168596] alg: No test for adler32 (adler32-zlib) [ 9993.185451] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 9993.515936] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9995.232229] Key type lgssc registered [ 9996.525443] Lustre: Echo OBD driver; http://www.lustre.org/ [10034.645725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10034.730652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.083679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10036.104486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10036.169616] Lustre: lustre-MDT0000: new disk, initializing [10036.267200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10036.279043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10040.435333] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10051.983139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.115662] Lustre: 342334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10052.168960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10052.174186] Lustre: Skipped 1 previous similar message [10052.228892] Lustre: lustre-MDT0001: new disk, initializing [10052.333621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10052.405372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10052.425236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10057.548881] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10066.729373] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10072.441957] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10080.273445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10080.763572] Lustre: lustre-OST0000: new disk, initializing [10080.790744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10080.802190] Lustre: 344302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10081.118433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10090.048611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10090.053674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10090.160275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10090.364709] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10101.845524] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10107.936277] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10108.152840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10112.882423] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10113.072191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10115.557680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10115.570322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.588129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.066446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10116.071026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.198412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.204277] Lustre: Skipped 2 previous similar messages [10121.412588] Lustre: server umount lustre-OST0000 complete [10126.311864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10126.326954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10126.334520] Lustre: Skipped 1 previous similar message [10131.377416] Lustre: server umount lustre-MDT0000 complete [10135.690107] LustreError: 344313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319732 with bad export cookie 469612666996916181 [10135.697844] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10135.698774] LustreError: 344313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10136.079456] Lustre: server umount lustre-MDT0001 complete [10147.659515] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:29:02 (1788319742) [10157.387576] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:29:13 (1788319753) [10166.412805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.956452] LustreError: 347061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10166.978635] LustreError: 347061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10167.070362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10171.876170] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10172.388978] LustreError: 347062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.407150] LustreError: 347062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10177.506634] LustreError: 347061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.080841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.419614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10184.644605] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10191.915817] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10196.471312] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10202.221058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10202.508518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10208.423399] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10216.290328] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10224.809398] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10226.760756] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10228.401183] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10230.220441] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10231.954647] Lustre: Failing over lustre-MDT0000 [10232.286642] Lustre: server umount lustre-MDT0000 complete [10233.316919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10233.323306] LustreError: 347057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.329547] Lustre: Skipped 2 previous similar messages [10233.353752] LustreError: 347057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10238.435456] LustreError: 347061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10238.460902] LustreError: 347061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10248.676113] LustreError: 347056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.689962] LustreError: 347056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10249.697202] Lustre: 340560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788319830/real 1788319830] req@00000000712e7924 x1875189002474752/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788319846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10249.715496] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10250.079044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.913150] LustreError: 340556:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000030562c8 x1875189002480256/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10259.302494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10259.383367] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10261.133535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10263.659835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10264.580670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10264.618299] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10264.680330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10273.790930] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10275.220942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10279.027844] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10281.303567] Lustre: Failing over lustre-MDT0000 [10281.674267] LustreError: 347056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10281.689728] LustreError: 347056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10281.795893] Lustre: server umount lustre-MDT0000 complete [10285.025500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10285.028126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.047990] Lustre: Skipped 2 previous similar messages [10301.415277] Lustre: 340559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788319882/real 1788319882] req@0000000003f643c7 x1875189002497024/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788319898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10301.457213] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10301.634333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10311.661796] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x684663b86600a78 [10311.674488] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [10311.691085] Lustre: Skipped 2 previous similar messages [10312.042269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10312.118478] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10312.338616] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10317.343237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10317.380710] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10317.473927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10317.607783] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10327.548966] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10329.266539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10337.763459] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10337.766439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10337.766523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10337.787397] Lustre: Skipped 2 previous similar messages [10342.881898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10342.886183] Lustre: Skipped 4 previous similar messages [10343.173844] Lustre: server umount lustre-MDT0000 complete [10347.308349] LustreError: 347043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788319944 with bad export cookie 469612666996918904 [10347.320483] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10347.325308] LustreError: 347043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10347.660799] Lustre: server umount lustre-MDT0001 complete [10362.693315] Lustre: server umount lustre-OST0000 complete [10379.314795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.687626] LustreError: 353405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.704345] LustreError: 353405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10379.765756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10383.035106] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10391.260479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.709326] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10403.285859] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10409.724975] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10417.609922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10424.200525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10425.213803] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10434.162793] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10444.772209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10444.783909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10444.796380] Lustre: Skipped 1 previous similar message [10444.811406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10449.865581] Lustre: server umount lustre-OST0000 complete [10459.865494] Lustre: server umount lustre-MDT0000 complete [10462.177481] LustreError: 355176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.222399] LustreError: 355176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10464.933375] LustreError: 353386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320062 with bad export cookie 469612666996919653 [10464.939232] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10464.943054] LustreError: 353386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10465.325205] Lustre: server umount lustre-MDT0001 complete [10477.566722] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10481.494895] Key type lgssc unregistered [10481.810956] LNet: 356942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10481.824700] LNetError: 356942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10481.846856] LNet: Removed LNI 192.168.204.145@tcp [10482.759065] Key type .llcrypt unregistered [10482.761703] Key type ._llcrypt unregistered [10508.820241] Key type ._llcrypt registered [10508.823212] Key type .llcrypt registered [10508.937767] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10509.998911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10510.145407] alg: No test for adler32 (adler32-zlib) [10511.150212] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [10511.332756] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10513.018550] Key type lgssc registered [10514.477148] Lustre: Echo OBD driver; http://www.lustre.org/ [10522.896462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10522.907826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10524.416229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10528.649397] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10533.860742] LustreError: 358459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.893761] LustreError: 358459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10537.003924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10537.439711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10541.963586] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10548.920540] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10555.253787] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10563.151870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.711404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10568.845336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10570.896713] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10578.633678] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10593.766068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10593.771508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10593.790764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10597.861242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10597.865660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.576454] Lustre: server umount lustre-MDT0000 complete [10602.980611] LustreError: 358453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.993380] LustreError: 358453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10603.116058] LustreError: 358438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320200 with bad export cookie 14662363489956438075 [10603.124285] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10603.128204] LustreError: 358438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10603.395534] Lustre: server umount lustre-MDT0001 complete [10616.189115] Lustre: server umount lustre-OST0000 complete [10629.284397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.748164] LustreError: 362191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.885861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10635.211711] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10635.240587] LustreError: 362192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.335378] LustreError: 362191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10642.880560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10643.312251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10647.789762] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10655.542031] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10661.222948] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10667.475730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10667.823792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10673.979876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10674.793827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10683.434081] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10699.753948] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10699.758649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10699.762344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10699.762353] Lustre: Skipped 2 previous similar messages [10699.780945] Lustre: Skipped 2 previous similar messages [10702.185465] Lustre: server umount lustre-MDT0000 complete [10704.875023] LustreError: 362920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10704.893818] LustreError: 362920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10705.495466] LustreError: 362927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320302 with bad export cookie 14662363489956439188 [10705.497029] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10705.511938] LustreError: 362927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10705.725168] Lustre: server umount lustre-MDT0001 complete [10719.755016] Lustre: server umount lustre-OST0000 complete [10736.513896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.988496] LustreError: 366103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10737.100700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10741.081388] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10749.343089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.339821] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10764.784438] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10770.930559] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10778.119293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10778.626902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10778.633985] Lustre: Skipped 1 previous similar message [10782.715301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10785.661720] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10793.825332] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10803.172854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10803.183783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10803.197619] Lustre: Skipped 1 previous similar message [10803.202834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10803.214805] Lustre: Skipped 2 previous similar messages [10805.725793] Lustre: server umount lustre-OST0000 complete [10811.363745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10811.371108] LustreError: Skipped 1 previous similar message [10811.374676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.385433] Lustre: Skipped 1 previous similar message [10811.390261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10811.394440] Lustre: Skipped 2 previous similar messages [10816.023463] Lustre: server umount lustre-MDT0000 complete [10819.576199] LustreError: 366104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10819.596480] LustreError: 366104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10819.940458] LustreError: 366085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320417 with bad export cookie 14662363489956440210 [10819.949787] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10819.959929] LustreError: 366085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10820.360986] Lustre: server umount lustre-MDT0001 complete [10831.893380] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10835.277954] Key type lgssc unregistered [10835.629241] LNet: 369638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10835.643484] LNetError: 369638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10835.663205] LNet: Removed LNI 192.168.204.145@tcp [10836.590629] Key type .llcrypt unregistered [10836.593809] Key type ._llcrypt unregistered [10863.142885] Key type ._llcrypt registered [10863.148034] Key type .llcrypt registered [10863.384178] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10864.919187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10864.972878] alg: No test for adler32 (adler32-zlib) [10866.119592] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [10866.383266] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10868.080230] Key type lgssc registered [10869.291143] Lustre: Echo OBD driver; http://www.lustre.org/ [10876.943905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10876.973901] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.984680] LustreError: 371095:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10877.011681] LustreError: 371095:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10877.024415] LustreError: 371095:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10877.035239] LustreError: 371095:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10877.042978] LustreError: 371095:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10884.874345] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.888952] LustreError: 371492:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10884.909953] LustreError: 371492:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10884.920412] LustreError: 371492:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10884.930145] LustreError: 371492:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10884.937963] LustreError: 371492:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10906.839257] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10910.182289] Key type lgssc unregistered [10910.491849] LNet: 372499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10910.511849] LNetError: 372499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10910.532586] LNet: Removed LNI 192.168.204.145@tcp [10911.632454] Key type .llcrypt unregistered [10911.637403] Key type ._llcrypt unregistered [10931.931214] Key type ._llcrypt registered [10931.944212] Key type .llcrypt registered [10932.104235] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [10947.817131] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10949.112363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10949.171777] alg: No test for adler32 (adler32-zlib) [10950.311689] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [10950.638816] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10952.328410] Key type lgssc registered [10953.574361] Lustre: Echo OBD driver; http://www.lustre.org/ [10988.594807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10988.625353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.848760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10989.887615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10989.950668] Lustre: lustre-MDT0000: new disk, initializing [10990.002231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10990.014352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10994.206855] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11003.628783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.807604] Lustre: 375655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11003.902858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11003.910919] Lustre: Skipped 1 previous similar message [11004.062659] Lustre: lustre-MDT0001: new disk, initializing [11004.182627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11004.224954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11004.239862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11009.091301] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11018.829329] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11024.920942] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11031.769874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11032.113654] Lustre: lustre-OST0000: new disk, initializing [11032.117106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11032.125883] Lustre: 377623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11032.243893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11039.316900] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11040.319131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11040.329908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11040.527753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11049.128075] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11054.033949] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11054.249608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11059.432167] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11059.662949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11066.345275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11066.352740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11066.365560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11066.378369] Lustre: Skipped 1 previous similar message [11066.392211] Lustre: Skipped 1 previous similar message [11067.787294] Lustre: server umount lustre-OST0000 complete [11076.065702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11076.071944] LustreError: Skipped 1 previous similar message [11076.074333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.089587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11077.941975] Lustre: server umount lustre-MDT0000 complete [11081.745804] LustreError: 375647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320678 with bad export cookie 1639411527869741201 [11081.761767] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11081.764779] LustreError: 375647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11082.167509] Lustre: server umount lustre-MDT0001 complete [11091.289628] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11093.010781] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:44:48 (1788320688) [11100.478907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.029552] LustreError: 380084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11101.053631] LustreError: 380084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11101.120991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11105.640351] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11106.274951] LustreError: 380085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.401362] LustreError: 380084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.801099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11114.066656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11118.937528] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11127.904299] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11133.747403] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11141.073523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11141.564594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11148.838291] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11156.718863] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11163.185879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.393186] Lustre: lustre-OST0001: new disk, initializing [11163.401566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11163.408293] Lustre: 382923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11163.496344] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11169.948474] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11172.979484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11172.990536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11173.157191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11180.442845] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11187.376622] Lustre: lustre-OST0001: Not available for connect from 192.168.204.45@tcp (stopping) [11187.389336] Lustre: Skipped 1 previous similar message [11188.706107] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11188.714735] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11188.719919] LustreError: Skipped 1 previous similar message [11188.754920] Lustre: Skipped 2 previous similar messages [11190.483990] Lustre: server umount lustre-OST0001 complete [11192.477329] LustreError: 381907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.959471] LustreError: 381906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.974146] LustreError: 381906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11209.194439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11209.196451] LustreError: 381906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.197368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11209.197373] Lustre: Skipped 2 previous similar messages [11209.215987] Lustre: Skipped 1 previous similar message [11209.250499] LustreError: 381906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11214.317447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11214.330978] Lustre: Skipped 2 previous similar messages [11223.008299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11223.134492] Lustre: server umount lustre-OST0000 complete [11229.666782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11229.679688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11229.685186] Lustre: Skipped 3 previous similar messages [11233.624059] Lustre: server umount lustre-MDT0000 complete [11234.793415] LustreError: 380085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11234.830513] LustreError: 380085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11237.570373] LustreError: 382922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320834 with bad export cookie 1639411527869742083 [11237.586673] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11237.596613] LustreError: 382922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11244.308414] Lustre: server umount lustre-MDT0001 complete [11257.468239] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11260.690760] Key type lgssc unregistered [11261.006780] LNet: 384915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11261.014502] LNetError: 384915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11261.029608] LNet: Removed LNI 192.168.204.145@tcp [11262.054336] Key type .llcrypt unregistered [11262.060909] Key type ._llcrypt unregistered [11315.195825] Key type ._llcrypt registered [11315.200347] Key type .llcrypt registered [11315.333977] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11316.307299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11316.348464] alg: No test for adler32 (adler32-zlib) [11317.445163] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [11317.728436] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11319.376176] Key type lgssc registered [11320.727498] Lustre: Echo OBD driver; http://www.lustre.org/ [11327.880794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11327.897253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.144161] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11329.166230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11329.314223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11333.404371] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11341.106801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.183988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11341.231218] Lustre: 387714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11341.602657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11346.462617] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11353.857947] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11359.498996] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11362.277977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.282046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11362.313488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.333631] Lustre: Skipped 1 previous similar message [11367.394066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.397855] Lustre: Skipped 1 previous similar message [11367.925293] Lustre: server umount lustre-MDT0000 complete [11371.436384] LustreError: 388456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788320968 with bad export cookie 15535913894286342407 [11371.437590] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11371.450033] LustreError: 388456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11371.754545] Lustre: server umount lustre-MDT0001 complete [11380.079060] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:49:35 (1788320975) [11385.741588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11386.272207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11390.384542] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11391.461355] LustreError: 390228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11391.473149] LustreError: 390228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11396.586165] LustreError: 390227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11398.631236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.045958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11403.896670] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11411.396818] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11416.087813] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11423.847735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11424.160818] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11424.503056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11432.329540] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11441.011303] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11448.924036] Lustre: Permanently deactivating lustre-OST0000 [11448.937915] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11457.070926] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11458.722767] LustreError: 393212:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11458.731869] LustreError: 393212:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11458.736908] LustreError: 393212:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11462.440399] LustreError: 393261:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11462.445263] LustreError: 393261:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11466.954858] Lustre: server umount lustre-OST0000 complete [11469.867132] LustreError: 393510:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11469.874468] LustreError: 393510:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11475.941482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11475.952524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.977332] Lustre: Skipped 1 previous similar message [11475.982484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11481.571135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11481.582596] Lustre: Skipped 3 previous similar messages [11482.771512] Lustre: server umount lustre-MDT0000 complete [11486.691171] LustreError: 392069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11486.720072] LustreError: 392069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11486.940409] LustreError: 390958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321084 with bad export cookie 15535913894286342988 [11486.942680] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11486.950391] LustreError: 390958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11497.722096] Lustre: server umount lustre-MDT0001 complete [11506.866972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11512.197839] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11515.186749] LustreError: 394675:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11521.033608] Lustre: 395014:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.145@tcp [11523.247991] Lustre: server umount MGS complete [11533.369049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11533.805417] LustreError: 395738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.844563] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11533.855158] Lustre: Skipped 1 previous similar message [11533.950737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.493951] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11539.438679] LustreError: 395739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11546.924032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11547.222236] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11552.968566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11560.824123] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11566.497685] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11568.449893] Lustre: Permanently reactivating lustre-OST0000 [11568.463957] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11568.468243] Lustre: Skipped 2 previous similar messages [11595.234816] LustreError: 397491:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11602.892844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11603.336687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11603.342680] Lustre: Skipped 1 previous similar message [11610.357835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11617.964440] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11625.638723] Lustre: server umount lustre-OST0000 complete [11629.556119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11629.573955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11629.586793] Lustre: Skipped 1 previous similar message [11629.595735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.331599] Lustre: server umount lustre-MDT0000 complete [11641.315303] LustreError: 397577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.328331] LustreError: 397577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11642.341254] Lustre: 387039:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788321223/real 1788321223] req@00000000d81a9892 x1875190390816384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788321239 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11642.362152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11642.370698] Lustre: Skipped 1 previous similar message [11643.004846] LustreError: 397314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321240 with bad export cookie 15535913894286344038 [11643.011740] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11643.013747] LustreError: 397314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11643.320757] Lustre: server umount lustre-MDT0001 complete [11653.945770] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11656.626513] Key type lgssc unregistered [11657.055272] LNet: 400105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11657.068724] LNetError: 400105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11658.087445] LNet: Removed LNI 192.168.204.145@tcp [11659.043156] Key type .llcrypt unregistered [11659.048551] Key type ._llcrypt unregistered [11679.249112] Key type ._llcrypt registered [11679.250430] Key type .llcrypt registered [11679.391390] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [11697.798521] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11699.603324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11699.638569] alg: No test for adler32 (adler32-zlib) [11700.992562] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [11701.273799] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11702.960233] Key type lgssc registered [11704.476496] Lustre: Echo OBD driver; http://www.lustre.org/ [11744.823473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11744.867031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.226338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11746.284272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11746.415636] Lustre: lustre-MDT0000: new disk, initializing [11746.564211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11746.662185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11751.654458] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11763.101875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11763.329676] Lustre: 403263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11763.440991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11763.446073] Lustre: Skipped 1 previous similar message [11763.555210] Lustre: lustre-MDT0001: new disk, initializing [11763.663250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11763.725025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11763.755247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11768.054355] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11777.434240] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11782.587889] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11788.793667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.024589] Lustre: lustre-OST0000: new disk, initializing [11789.027790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11789.033328] Lustre: 405233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11789.107225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11795.260047] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11796.564134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11796.579475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11796.697715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11804.505445] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11809.169533] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11809.378400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11814.292987] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11814.604355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11817.442031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11817.455707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11817.470892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11817.478070] Lustre: Skipped 1 previous similar message [11817.491753] Lustre: Skipped 1 previous similar message [11822.572452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11822.579107] Lustre: Skipped 1 previous similar message [11827.686427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.697239] Lustre: Skipped 1 previous similar message [11831.266764] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11831.544967] Lustre: server umount lustre-OST0000 complete [11835.362331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11835.373565] LustreError: Skipped 1 previous similar message [11835.379012] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.400368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11837.921812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.736976] Lustre: server umount lustre-MDT0000 complete [11845.629418] LustreError: 403255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321442 with bad export cookie 2162680641022220788 [11845.638557] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11845.653050] LustreError: 403255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11846.000774] Lustre: server umount lustre-MDT0001 complete [11855.146361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11857.300359] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:57:32 (1788321452) [11864.786052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.387936] LustreError: 407697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.404117] LustreError: 407697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11865.438677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11870.033748] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11870.689936] LustreError: 407698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.814844] LustreError: 407697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11878.486321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.798431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11882.375808] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11891.194080] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11897.394702] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11905.931430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11906.258279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11912.114366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11919.757785] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11922.039792] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11924.806622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11932.137950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11932.150674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11932.167248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11932.178754] Lustre: Skipped 2 previous similar messages [11934.186319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11936.470838] Lustre: server umount lustre-OST0000 complete [11940.322396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11940.333142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11940.341944] Lustre: Skipped 1 previous similar message [11940.347764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11940.354435] Lustre: Skipped 2 previous similar messages [11946.530293] Lustre: server umount lustre-MDT0000 complete [11949.543361] LustreError: 408426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11949.565297] LustreError: 408426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11950.033447] LustreError: 407679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321547 with bad export cookie 2162680641022221670 [11950.037950] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11950.044245] LustreError: 407679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11950.332723] Lustre: server umount lustre-MDT0001 complete [11960.932386] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11963.682699] Key type lgssc unregistered [11963.968520] LNet: 411423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11963.979560] LNetError: 411423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11963.993066] LNet: Removed LNI 192.168.204.145@tcp [11964.771706] Key type .llcrypt unregistered [11964.777687] Key type ._llcrypt unregistered [11985.877908] Key type ._llcrypt registered [11985.880682] Key type .llcrypt registered [11985.982179] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11986.410839] Key type .llcrypt unregistered [11986.415915] Key type ._llcrypt unregistered [12006.707295] Key type ._llcrypt registered [12006.709593] Key type .llcrypt registered [12006.832494] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12008.155218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12008.202192] alg: No test for adler32 (adler32-zlib) [12009.404025] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [12009.748698] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12011.408169] Key type lgssc registered [12012.443146] Lustre: Echo OBD driver; http://www.lustre.org/ [12020.304993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12020.313924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.683418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12026.270385] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12030.959938] LustreError: 413645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12030.981466] LustreError: 413645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12032.894603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12033.226314] LustreError: 413644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12033.247504] LustreError: 413644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12033.255904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12038.643524] LustreError: 413640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12038.695476] LustreError: 413640:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12040.506644] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12043.751955] LustreError: 413644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12043.772634] LustreError: 413644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12046.839441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12048.694791] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12051.942556] LustreError: 413645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12051.960391] LustreError: 413645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12056.077398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.382843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12061.581392] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12070.450368] Lustre: server umount lustre-OST0000 complete [12077.033403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.034882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12077.043717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.046505] Lustre: Skipped 1 previous similar message [12082.149732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12082.164238] Lustre: Skipped 2 previous similar messages [12084.300341] Lustre: server umount lustre-MDT0000 complete [12087.272478] LustreError: 415638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.290472] LustreError: 415638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12088.039830] LustreError: 413624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321685 with bad export cookie 13050140209071383518 [12088.057347] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12098.696187] Lustre: server umount lustre-MDT0001 complete [12110.129328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12113.334754] Key type lgssc unregistered [12113.754387] LNet: 416808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12113.768778] LNetError: 416808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12113.791817] LNet: Removed LNI 192.168.204.145@tcp [12114.932297] Key type .llcrypt unregistered [12114.935651] Key type ._llcrypt unregistered [12141.345351] Key type ._llcrypt registered [12141.353591] Key type .llcrypt registered [12141.475878] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12142.478224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12142.617142] alg: No test for adler32 (adler32-zlib) [12143.678657] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [12144.033592] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12145.728255] Key type lgssc registered [12146.691220] Lustre: Echo OBD driver; http://www.lustre.org/ [12154.478612] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12154.491674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12170.144905] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12175.264448] LustreError: 418200:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [12201.953527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12209.888584] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12219.433879] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12227.346185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.087904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.167804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12233.192581] LustreError: 418226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12233.206043] LustreError: 418226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12233.768829] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12236.264973] LustreError: 419237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12236.277713] LustreError: 419237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12241.384023] LustreError: 419237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12241.401144] LustreError: 419237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12241.497228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12241.949322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12246.491660] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12253.615058] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12259.148644] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12267.490152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12267.501118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.502805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12267.509421] Lustre: Skipped 1 previous similar message [12268.218127] Lustre: server umount lustre-OST0000 complete [12272.611102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12272.611505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12272.624788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12272.640525] Lustre: Skipped 1 previous similar message [12272.649395] Lustre: Skipped 2 previous similar messages [12277.730243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12277.737098] Lustre: Skipped 1 previous similar message [12278.333282] Lustre: server umount lustre-MDT0000 complete [12281.720066] LustreError: 418206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321878 with bad export cookie 3101016936693791166 [12281.734589] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12281.736323] LustreError: 418206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12281.968341] Lustre: server umount lustre-MDT0001 complete [12292.433405] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12295.212435] Key type lgssc unregistered [12295.564889] LNet: 421772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12295.574180] LNetError: 421772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12295.599275] LNet: Removed LNI 192.168.204.145@tcp [12296.506199] Key type .llcrypt unregistered [12296.510426] Key type ._llcrypt unregistered [12321.573861] Key type ._llcrypt registered [12321.578675] Key type .llcrypt registered [12321.732640] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12323.402875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12323.467810] alg: No test for adler32 (adler32-zlib) [12324.505574] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [12324.805439] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12326.512406] Key type lgssc registered [12327.749443] Lustre: Echo OBD driver; http://www.lustre.org/ [12335.389975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12335.429628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.799369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12341.013910] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12346.340116] LustreError: 423190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.366648] LustreError: 423190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12349.122658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12349.496691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12353.494854] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12361.257461] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12367.229373] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12374.241891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12374.523072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12380.337372] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12381.705719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12387.885624] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12394.084049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12394.108151] Lustre: Skipped 1 previous similar message [12394.108941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12394.115091] Lustre: Skipped 2 previous similar messages [12394.390501] Lustre: server umount lustre-MDT0000 complete [12395.178545] LustreError: 423185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12397.428592] LustreError: 423170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1788321994 with bad export cookie 607487275873340736 [12397.477572] LustreError: 423170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12399.588916] LustreError: 423184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12399.612868] LustreError: 423184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12399.810390] LustreError: 425033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788321997 with bad export cookie 607487275873339847 [12399.833182] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12399.839702] LustreError: 425033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12402.330620] Lustre: server umount lustre-OST0000 complete [12404.714412] LustreError: 424233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12404.740931] LustreError: 424233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12409.827648] LustreError: 423185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12409.846798] LustreError: 423185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12410.528578] LustreError: 423918:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009ea34bb2 x1875191446667008/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12420.798683] Lustre: server umount lustre-MDT0001 complete [12432.592632] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12435.522513] Key type lgssc unregistered [12435.860091] LNet: 426879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12435.870931] LNetError: 426879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12435.896308] LNet: Removed LNI 192.168.204.145@tcp [12436.918190] Key type .llcrypt unregistered [12436.920477] Key type ._llcrypt unregistered [12465.380952] Key type ._llcrypt registered [12465.384412] Key type .llcrypt registered [12465.595933] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12467.268297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12467.373054] alg: No test for adler32 (adler32-zlib) [12468.505793] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [12468.811772] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12470.505289] Key type lgssc registered [12471.490717] Lustre: Echo OBD driver; http://www.lustre.org/ [12479.259719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12479.303262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12480.695614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12485.691736] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12490.230961] LustreError: 428298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.261777] LustreError: 428298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12495.205655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12495.331662] LustreError: 428297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.626524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12499.623022] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12507.851423] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12512.584818] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12519.895458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12520.145770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12525.379185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12528.373456] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12537.021091] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12545.201981] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [12546.019119] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.035746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12546.036221] Lustre: Skipped 1 previous similar message [12546.056215] Lustre: Skipped 1 previous similar message [12548.423806] Lustre: server umount lustre-MDT0001 complete [12550.318992] LustreError: 429886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.259768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12556.262519] LustreError: 428297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12556.297473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12556.312177] LustreError: 428297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12561.391159] LustreError: 430119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.406054] LustreError: 430119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12561.582374] Lustre: server umount lustre-OST0000 complete [12565.910748] Lustre: server umount lustre-MDT0000 complete [12580.238941] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12583.693603] Key type lgssc unregistered [12584.061642] LNet: 431986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12584.083050] LNetError: 431986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12584.103322] LNet: Removed LNI 192.168.204.145@tcp [12585.110292] Key type .llcrypt unregistered [12585.113417] Key type ._llcrypt unregistered [12610.306946] Key type ._llcrypt registered [12610.309168] Key type .llcrypt registered [12610.405216] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [12626.562574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12627.731323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12627.780073] alg: No test for adler32 (adler32-zlib) [12629.020166] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [12629.396862] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12631.072188] Key type lgssc registered [12632.641307] Lustre: Echo OBD driver; http://www.lustre.org/ [12674.871092] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12692.360045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12692.397097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12693.734525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12693.805251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12693.961879] Lustre: lustre-MDT0000: new disk, initializing [12694.048221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12694.073443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12699.698988] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12716.287780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12716.659231] Lustre: lustre-OST0000: new disk, initializing [12716.662304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12716.669802] Lustre: Skipped 1 previous similar message [12716.676924] Lustre: 437020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12716.765847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12723.548344] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12724.846151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12724.861475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12724.893777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12733.698125] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12749.132662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12749.227106] Lustre: 436003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12749.289862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12749.348357] Lustre: lustre-MDT0001: new disk, initializing [12749.482676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12749.541145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12749.556576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12754.165574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12767.781484] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12768.137697] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12774.340361] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12774.644723] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12780.727474] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [12782.066142] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12782.071339] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12782.071958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12782.088869] Lustre: Skipped 1 previous similar message [12785.808239] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [12785.818399] Lustre: Skipped 1 previous similar message [12787.123724] Lustre: server umount lustre-MDT0001 complete [12792.292338] LustreError: 436010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12792.318640] LustreError: 436010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12797.426764] LustreError: 436011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.454386] LustreError: 436011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12802.538840] LustreError: 436010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12802.562951] LustreError: 436010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12807.651893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12807.658654] LustreError: 436014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.673213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12812.784186] LustreError: 436015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12812.784756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12812.982030] Lustre: server umount lustre-OST0000 complete [12824.420666] Lustre: server umount lustre-MDT0000 complete [12839.048977] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12842.273168] Key type lgssc unregistered [12842.538203] LNet: 440774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12842.561174] LNetError: 440774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12842.585282] LNet: Removed LNI 192.168.204.145@tcp [12843.632207] Key type .llcrypt unregistered [12843.635876] Key type ._llcrypt unregistered [12886.257474] Key type ._llcrypt registered [12886.262737] Key type .llcrypt registered [12886.413430] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12886.970427] Key type .llcrypt unregistered [12886.982101] Key type ._llcrypt unregistered [12906.853156] Key type ._llcrypt registered [12906.857322] Key type .llcrypt registered [12907.012096] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12908.398583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12908.432063] alg: No test for adler32 (adler32-zlib) [12909.670506] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [12910.031157] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12911.812115] Key type lgssc registered [12913.445835] Lustre: Echo OBD driver; http://www.lustre.org/ [12944.985801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12945.024090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12946.358342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12946.447678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12946.576146] Lustre: lustre-MDT0000: new disk, initializing [12946.688868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12946.722966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12952.077705] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12964.163467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12964.334429] Lustre: 444165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12964.539936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12964.551049] Lustre: Skipped 1 previous similar message [12964.737592] Lustre: lustre-MDT0001: new disk, initializing [12964.831231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12964.886805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12964.907985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12970.282903] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12980.525191] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12986.864270] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12994.677607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12995.001455] Lustre: lustre-OST0000: new disk, initializing [12995.004609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12995.009834] Lustre: 446133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12995.112439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13001.959776] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13002.313461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13002.321299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13002.442860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13011.858635] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13018.085347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13018.095635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13018.099463] Lustre: Skipped 1 previous similar message [13018.122421] Lustre: Skipped 2 previous similar messages [13023.207188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13023.214740] Lustre: Skipped 2 previous similar messages [13024.344470] Lustre: server umount lustre-MDT0000 complete [13028.000410] LustreError: 444157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788322625 with bad export cookie 4896077143372972285 [13028.007310] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.011696] LustreError: 444157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13028.409950] Lustre: server umount lustre-MDT0001 complete [13042.425040] Lustre: server umount lustre-OST0000 complete [13053.778427] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:17:29 (1788322649) [13061.340059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13061.941739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13066.957856] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13067.237145] LustreError: 448160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13067.269692] LustreError: 448160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13072.364289] LustreError: 448161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13075.434750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13075.741909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13080.425545] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13087.614915] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13092.768409] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13112.437791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13112.591126] Lustre: Found index 0 for lustre-OST0000, updating log [13112.822293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13118.792523] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13127.366098] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13133.369894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13138.410879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13138.418161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13138.421102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13138.445030] Lustre: Skipped 2 previous similar messages [13139.310805] Lustre: server umount lustre-OST0000 complete [13156.390339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13156.397253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13156.403557] Lustre: Skipped 1 previous similar message [13156.409806] Lustre: Skipped 1 previous similar message [13161.441104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13161.460913] Lustre: Skipped 1 previous similar message [13163.761491] Lustre: server umount lustre-MDT0000 complete [13166.586661] LustreError: 449600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13166.607298] LustreError: 449600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13168.137483] LustreError: 448141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788322765 with bad export cookie 4896077143372973678 [13168.139675] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13168.154316] LustreError: 448141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13168.583266] Lustre: server umount lustre-MDT0001 complete [13173.282889] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [13177.773170] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:19:33 (1788322773) [13795.540430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13795.941994] LustreError: 454570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13796.023268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13801.406539] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13801.454645] LustreError: 454571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13801.475978] LustreError: 454571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13806.565320] LustreError: 454570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.671095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13812.706340] LustreError: 454571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.728919] LustreError: 454571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13813.102916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13819.058976] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13828.438680] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13834.819689] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13851.168242] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [13854.843731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13855.253687] Lustre: Found index 0 for lustre-OST0000, updating log [13855.682840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13863.307436] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13871.808180] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13874.723725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13874.738093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13879.826342] Lustre: server umount lustre-OST0000 complete [13884.066535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13884.075505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13884.103944] Lustre: Skipped 1 previous similar message [13886.178161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13886.181872] Lustre: Skipped 4 previous similar messages [13889.392822] Lustre: server umount lustre-MDT0000 complete [13890.283566] LustreError: 454565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13890.310382] LustreError: 454565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13893.026666] LustreError: 456435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788323490 with bad export cookie 4896077143372974756 [13893.029038] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13893.035211] LustreError: 456435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13893.568532] Lustre: server umount lustre-MDT0001 complete [13911.585753] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [13925.896191] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [13937.148327] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13977.277829] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:32:52 (1788323572) [13997.167445] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14000.899606] Key type lgssc unregistered [14001.421720] LNet: 461188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14001.453652] LNetError: 461188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14116.219777] LNet: Removed LNI 192.168.204.145@tcp [14117.172613] Key type .llcrypt unregistered [14117.176222] Key type ._llcrypt unregistered [14133.599528] Key type ._llcrypt registered [14133.605203] Key type .llcrypt registered [14133.743659] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14135.110982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14135.172483] alg: No test for adler32 (adler32-zlib) [14136.369522] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [14136.686261] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14138.448194] Key type lgssc registered [14139.930425] Lustre: Echo OBD driver; http://www.lustre.org/ [14144.557604] LNet: Added LNI 192.168.204.145@tcp100 [8/256/0/180] [14145.429875] LNet: Added LNI 192.168.204.145@tcp101 [8/256/0/180] [14146.316350] LNet: Added LNI 192.168.204.145@tcp102 [8/256/0/180] [14147.073946] LNet: Added LNI 192.168.204.145@tcp103 [8/256/0/180] [14147.917959] LNet: Added LNI 192.168.204.145@tcp104 [8/256/0/180] [14148.918718] LNet: Added LNI 192.168.204.145@tcp105 [8/256/0/180] [14149.709950] LNet: Added LNI 192.168.204.145@tcp106 [8/256/0/180] [14150.425029] LNet: Added LNI 192.168.204.145@tcp107 [8/256/0/180] [14151.160585] LNet: Added LNI 192.168.204.145@tcp108 [8/256/0/180] [14151.965588] LNet: Added LNI 192.168.204.145@tcp109 [8/256/0/180] [14152.895488] LNet: Added LNI 192.168.204.145@tcp110 [8/256/0/180] [14153.666094] LNet: Added LNI 192.168.204.145@tcp111 [8/256/0/180] [14154.576969] LNet: Added LNI 192.168.204.145@tcp112 [8/256/0/180] [14155.396223] LNet: Added LNI 192.168.204.145@tcp113 [8/256/0/180] [14156.310593] LNet: Added LNI 192.168.204.145@tcp114 [8/256/0/180] [14157.212739] LNet: Added LNI 192.168.204.145@tcp115 [8/256/0/180] [14158.015927] LNet: Added LNI 192.168.204.145@tcp116 [8/256/0/180] [14159.074966] LNet: Added LNI 192.168.204.145@tcp117 [8/256/0/180] [14159.916591] LNet: Added LNI 192.168.204.145@tcp118 [8/256/0/180] [14160.779149] LNet: Added LNI 192.168.204.145@tcp119 [8/256/0/180] [14161.660786] LNet: Added LNI 192.168.204.145@tcp120 [8/256/0/180] [14162.517084] LNet: Added LNI 192.168.204.145@tcp121 [8/256/0/180] [14163.442960] LNet: Added LNI 192.168.204.145@tcp122 [8/256/0/180] [14164.190893] LNet: Added LNI 192.168.204.145@tcp123 [8/256/0/180] [14165.128328] LNet: Added LNI 192.168.204.145@tcp124 [8/256/0/180] [14165.982616] LNet: Added LNI 192.168.204.145@tcp125 [8/256/0/180] [14166.621757] LNet: Added LNI 192.168.204.145@tcp126 [8/256/0/180] [14167.369447] LNet: Added LNI 192.168.204.145@tcp127 [8/256/0/180] [14168.202065] LNet: Added LNI 192.168.204.145@tcp128 [8/256/0/180] [14169.105845] LNet: Added LNI 192.168.204.145@tcp129 [8/256/0/180] [14169.926683] LNet: Added LNI 192.168.204.145@tcp130 [8/256/0/180] [14170.824901] LNet: Added LNI 192.168.204.145@tcp131 [8/256/0/180] [14171.709176] LNet: Added LNI 192.168.204.145@tcp132 [8/256/0/180] [14172.549580] LNet: Added LNI 192.168.204.145@tcp133 [8/256/0/180] [14173.476853] LNet: Added LNI 192.168.204.145@tcp134 [8/256/0/180] [14174.327458] LNet: Added LNI 192.168.204.145@tcp135 [8/256/0/180] [14181.169684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14181.204745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14182.304642] Lustre: 464222:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14182.374332] LNetError: 461680:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14182.389429] LNetError: Protocol error connecting to 192.168.204.145@tcp117 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14182.397862] LNetError: Skipped 3 previous similar messages [14182.502366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14182.548434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14182.651161] Lustre: lustre-MDT0000: new disk, initializing [14182.781294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14182.798607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14182.819281] LNet: 5 local NIs in recovery (showing 5): 192.168.204.145@tcp123, 192.168.204.145@tcp124, 192.168.204.145@tcp125, 192.168.204.145@tcp128, 192.168.204.145@tcp129 [14183.458670] LNetError: 461681:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14183.473925] LNetError: 461681:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14183.480787] LNetError: Protocol error connecting to 192.168.204.145@tcp114 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14183.499290] LNetError: Skipped 5 previous similar messages [14185.889618] LNetError: 464251:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14185.903225] LNetError: 464251:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14185.911240] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14185.923881] LNetError: Skipped 5 previous similar messages [14186.497483] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14189.986795] LNetError: 461680:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14190.002532] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14197.277743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14197.471197] Lustre: 464238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14197.580757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14197.590541] Lustre: Skipped 1 previous similar message [14197.722506] Lustre: lustre-MDT0001: new disk, initializing [14197.761323] Lustre: 465039:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14197.767401] Lustre: 465039:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14197.847958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14197.882468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14197.894142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14202.120746] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14206.024190] Lustre: 465458:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14206.042632] Lustre: 465458:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14212.079716] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14213.539707] LNetError: 464242:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14213.555236] LNetError: Protocol error connecting to 192.168.204.145@tcp123 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14217.684717] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14221.669691] LNetError: 461681:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14221.682538] LNetError: 461681:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14221.690836] LNetError: Protocol error connecting to 192.168.204.145@tcp117 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14221.701402] LNetError: Skipped 3 previous similar messages [14224.531886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14224.983536] Lustre: lustre-OST0000: new disk, initializing [14224.996359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14225.017091] Lustre: 466214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14225.226198] Lustre: 466219:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14225.237288] Lustre: 466219:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14225.341251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14231.113323] Lustre: 466467:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.145@tcp: can't add NID 192.168.204.145@tcp130: rc = -75 [14231.121466] Lustre: 466467:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14231.194588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14231.208428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14231.460329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14233.063721] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14241.144800] LNetError: 461681:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14241.175794] LNetError: 461681:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14241.203087] LNetError: Protocol error connecting to 192.168.204.145@tcp121 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14241.247505] LNetError: Skipped 7 previous similar messages [14243.232276] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp123 [14245.002589] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14252.004297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14252.021500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14252.023360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14252.041530] Lustre: Skipped 1 previous similar message [14257.132411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14257.141014] Lustre: Skipped 1 previous similar message [14262.243532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14262.257480] Lustre: Skipped 1 previous similar message [14266.336148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14266.487950] Lustre: server umount lustre-OST0000 complete [14272.484128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14272.493132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14272.495776] Lustre: Skipped 2 previous similar messages [14272.517326] Lustre: Skipped 1 previous similar message [14277.607886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14277.621167] Lustre: Skipped 1 previous similar message [14278.099104] Lustre: server umount lustre-MDT0000 complete [14279.019410] LNetError: 461683:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.145 [14279.034887] LNetError: 461683:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14279.045957] LNetError: Protocol error connecting to 192.168.204.145@tcp121 at host 192.168.204.145:988: is it running a compatible version of Lustre? [14279.053969] LNetError: Skipped 10 previous similar messages [14281.936830] LustreError: 464231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788323879 with bad export cookie 2783811011312894032 [14281.949277] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14281.954188] LustreError: 464231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14282.568832] Lustre: server umount lustre-MDT0001 complete [14294.688302] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14298.241818] Key type lgssc unregistered [14298.698949] LNet: 468132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14298.713840] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14298.726405] LNet: Removed LNI 192.168.204.145@tcp [14299.745618] LNet: Removed LNI 192.168.204.145@tcp100 [14299.749457] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14299.774554] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14299.785162] LNet: Removed LNI 192.168.204.145@tcp101 [14300.832516] LNet: Removed LNI 192.168.204.145@tcp102 [14300.839038] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14300.854176] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14300.867659] LNet: Removed LNI 192.168.204.145@tcp103 [14301.920584] LNet: Removed LNI 192.168.204.145@tcp104 [14302.944409] LNet: Removed LNI 192.168.204.145@tcp105 [14302.948529] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14302.963952] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14302.977391] LNet: Removed LNI 192.168.204.145@tcp106 [14304.032210] LNet: Removed LNI 192.168.204.145@tcp107 [14305.056605] LNet: Removed LNI 192.168.204.145@tcp108 [14306.080284] LNet: Removed LNI 192.168.204.145@tcp109 [14307.106478] LNet: Removed LNI 192.168.204.145@tcp110 [14307.113668] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14307.122575] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14307.132799] LNet: Removed LNI 192.168.204.145@tcp111 [14308.192857] LNet: Removed LNI 192.168.204.145@tcp112 [14309.216774] LNet: Removed LNI 192.168.204.145@tcp113 [14310.240217] LNet: Removed LNI 192.168.204.145@tcp114 [14311.273890] LNet: Removed LNI 192.168.204.145@tcp115 [14312.288357] LNet: Removed LNI 192.168.204.145@tcp116 [14313.312188] LNet: Removed LNI 192.168.204.145@tcp117 [14314.337386] LNet: Removed LNI 192.168.204.145@tcp118 [14315.360154] LNet: Removed LNI 192.168.204.145@tcp119 [14315.364653] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14315.373299] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14315.382028] LNet: Removed LNI 192.168.204.145@tcp120 [14316.448154] LNet: Removed LNI 192.168.204.145@tcp121 [14317.472199] LNet: Removed LNI 192.168.204.145@tcp122 [14317.478703] LNet: Removed LNI 192.168.204.145@tcp123 [14318.500072] LNet: Removed LNI 192.168.204.145@tcp124 [14319.520168] LNet: Removed LNI 192.168.204.145@tcp125 [14320.544485] LNet: Removed LNI 192.168.204.145@tcp126 [14321.568303] LNet: Removed LNI 192.168.204.145@tcp127 [14322.592099] LNet: Removed LNI 192.168.204.145@tcp128 [14323.616165] LNet: Removed LNI 192.168.204.145@tcp129 [14323.626520] LNet: Removed LNI 192.168.204.145@tcp130 [14323.630452] LNet: Removed LNI 192.168.204.145@tcp131 [14323.634671] LNet: Removed LNI 192.168.204.145@tcp132 [14323.639935] LNet: Removed LNI 192.168.204.145@tcp133 [14323.642658] LNet: Removed LNI 192.168.204.145@tcp134 [14323.657741] LNet: Removed LNI 192.168.204.145@tcp135 [14324.395855] Key type .llcrypt unregistered [14324.398024] Key type ._llcrypt unregistered [14345.772891] Key type ._llcrypt registered [14345.775966] Key type .llcrypt registered [14345.888207] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [14361.283719] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14362.813631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14362.860896] alg: No test for adler32 (adler32-zlib) [14363.960251] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [14364.198753] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14365.888180] Key type lgssc registered [14367.271873] Lustre: Echo OBD driver; http://www.lustre.org/ [14402.178119] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:39:58 (1788323998) [14421.241274] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14424.293104] Key type lgssc unregistered [14424.805579] LNet: 472236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14424.824654] LNetError: 472236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14424.862197] LNet: Removed LNI 192.168.204.145@tcp [14426.046828] Key type .llcrypt unregistered [14426.051275] Key type ._llcrypt unregistered [14444.122912] Key type ._llcrypt registered [14444.125730] Key type .llcrypt registered [14444.339173] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14445.181143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14445.280400] alg: No test for adler32 (adler32-zlib) [14446.313134] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [14446.573826] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14448.248690] Key type lgssc registered [14449.613105] Lustre: Echo OBD driver; http://www.lustre.org/ [14459.641932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14459.664766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14461.046717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14461.099564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14461.184460] Lustre: lustre-MDT0000: new disk, initializing [14461.309125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14461.338246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14466.631531] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14476.626202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14476.730711] Lustre: 473468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14476.781981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14476.786829] Lustre: Skipped 1 previous similar message [14476.854305] Lustre: lustre-MDT0001: new disk, initializing [14476.962809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14476.988542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14476.996941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14480.629284] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14489.296650] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14495.140097] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14501.896114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14502.198704] Lustre: lustre-OST0000: new disk, initializing [14502.202257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14502.208978] Lustre: 475437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14502.271403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14509.184831] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14509.648564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14509.689802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14509.890184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14518.779469] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14522.224647] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lnet_if_list [14526.067117] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14526.086846] LNet: Added LNI 192.168.204.145@tcp10 [8/256/0/180] [14530.997428] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14531.030036] LNet: Added LNI 192.168.204.145@tcp11 [8/256/0/180] [14534.980215] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14535.021498] LNet: Added LNI 192.168.204.145@tcp12 [8/256/0/180] [14540.685551] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14540.726791] LNet: Added LNI 192.168.204.145@tcp13 [8/256/0/180] [14546.102852] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14546.165655] LNet: Added LNI 192.168.204.145@tcp14 [8/256/0/180] [14550.445662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14550.479091] LNet: Added LNI 192.168.204.145@tcp15 [8/256/0/180] [14555.303567] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14555.331086] LNet: Added LNI 192.168.204.145@tcp16 [8/256/0/180] [14560.073781] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14560.100437] LNet: Added LNI 192.168.204.145@tcp17 [8/256/0/180] [14564.947525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14564.976579] LNet: Added LNI 192.168.204.145@tcp18 [8/256/0/180] [14570.277595] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14570.314827] LNet: Added LNI 192.168.204.145@tcp19 [8/256/0/180] [14575.435959] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14575.459837] LNet: Added LNI 192.168.204.145@tcp20 [8/256/0/180] [14581.474622] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14581.515668] LNet: Added LNI 192.168.204.145@tcp21 [8/256/0/180] [14586.881166] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14586.907892] LNet: Added LNI 192.168.204.145@tcp22 [8/256/0/180] [14592.327800] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14592.376650] LNet: Added LNI 192.168.204.145@tcp23 [8/256/0/180] [14597.432341] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14597.461457] LNet: Added LNI 192.168.204.145@tcp24 [8/256/0/180] [14602.941923] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14602.980785] LNet: Added LNI 192.168.204.145@tcp25 [8/256/0/180] [14608.761607] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14608.789687] LNet: Added LNI 192.168.204.145@tcp26 [8/256/0/180] [14614.680649] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14614.707486] LNet: Added LNI 192.168.204.145@tcp27 [8/256/0/180] [14620.785216] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14620.841241] LNet: Added LNI 192.168.204.145@tcp28 [8/256/0/180] [14626.099251] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14626.119459] LNet: Added LNI 192.168.204.145@tcp29 [8/256/0/180] [14631.356447] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14631.387628] LNet: Added LNI 192.168.204.145@tcp30 [8/256/0/180] [14637.551625] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14637.576842] LNet: Added LNI 192.168.204.145@tcp31 [8/256/0/180] [14643.696797] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14643.734977] LNet: Added LNI 192.168.204.145@tcp32 [8/256/0/180] [14650.502154] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14650.530164] LNet: Added LNI 192.168.204.145@tcp33 [8/256/0/180] [14657.840177] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14657.889630] LNet: Added LNI 192.168.204.145@tcp34 [8/256/0/180] [14664.038938] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14664.078707] LNet: Added LNI 192.168.204.145@tcp35 [8/256/0/180] [14670.068125] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14670.111483] LNet: Added LNI 192.168.204.145@tcp36 [8/256/0/180] [14679.545252] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14679.590440] LNet: Added LNI 192.168.204.145@tcp37 [8/256/0/180] [14689.599565] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14689.635944] LNet: Added LNI 192.168.204.145@tcp38 [8/256/0/180] [14696.248907] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14696.269586] LNet: Added LNI 192.168.204.145@tcp39 [8/256/0/180] [14702.038385] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14702.056263] LNet: Added LNI 192.168.204.145@tcp40 [8/256/0/180] [14709.639115] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14709.675418] LNet: Added LNI 192.168.204.145@tcp41 [8/256/0/180] [14718.864056] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14718.890962] LNet: Added LNI 192.168.204.145@tcp42 [8/256/0/180] [14727.820342] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14727.851254] LNet: Added LNI 192.168.204.145@tcp43 [8/256/0/180] [14734.568188] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14734.600382] LNet: Added LNI 192.168.204.145@tcp44 [8/256/0/180] [14741.566384] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14741.614574] LNet: Added LNI 192.168.204.145@tcp45 [8/256/0/180] [14750.301725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14750.322343] LNet: Added LNI 192.168.204.145@tcp46 [8/256/0/180] [14760.303447] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14760.342424] LNet: Added LNI 192.168.204.145@tcp47 [8/256/0/180] [14769.434454] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14769.464914] LNet: Added LNI 192.168.204.145@tcp48 [8/256/0/180] [14777.940688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14777.987287] LNet: Added LNI 192.168.204.145@tcp49 [8/256/0/180] [14786.150573] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14786.173825] LNet: Added LNI 192.168.204.145@tcp50 [8/256/0/180] [14807.013814] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14807.040951] Lustre: Skipped 2 previous similar messages [14807.042335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14807.065117] Lustre: Skipped 2 previous similar messages [14811.771958] Lustre: server umount lustre-MDT0000 complete [14815.101375] LustreError: 474260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788324412 with bad export cookie 3814378961920854006 [14815.101951] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14815.120777] LustreError: 474260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14815.495641] Lustre: server umount lustre-MDT0001 complete [14831.884971] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14833.379361] Lustre: 472736:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788324414/real 1788324414] req@00000000e315b628 x1875193671879296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788324430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14833.442110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14836.707415] Lustre: 472737:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788324417/real 1788324417] req@000000003a4fb8d2 x1875193671879424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788324433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14838.822238] Lustre: 472737:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788324419/real 1788324419] req@00000000ade393a3 x1875193671879552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788324435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14845.920058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14845.921054] Lustre: 472735:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788324427/real 1788324427] req@00000000081b7db7 x1875193671879936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788324443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14845.991309] Lustre: 472735:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14860.133185] LustreError: 472734:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a6155e2e x1875193671880704/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14860.522948] LustreError: 475428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14860.540568] LustreError: 475428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14860.594586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14862.630525] LustreError: 475429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14862.653512] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14863.650632] LustreError: 475428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14865.077965] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14868.967910] LustreError: 475429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14873.056832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14873.500525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14877.923389] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [14878.641696] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14884.562547] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14888.838249] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14904.310405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14904.316908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14904.337148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14904.361220] Lustre: Skipped 1 previous similar message [14904.373399] Lustre: Skipped 1 previous similar message [14909.412751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14910.464189] Lustre: server umount lustre-OST0000 complete [14914.530970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14914.531264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14914.564347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14914.566987] Lustre: Skipped 1 previous similar message [14919.651447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14919.665452] Lustre: Skipped 1 previous similar message [14920.655063] Lustre: server umount lustre-MDT0000 complete [14924.629273] LustreError: 474260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788324521 with bad export cookie 3814378961920856589 [14924.636642] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14924.638581] LustreError: 474260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14924.865797] Lustre: server umount lustre-MDT0001 complete [14936.910153] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [14940.185884] Key type lgssc unregistered [14940.723364] LNet: 502437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14940.744198] LNetError: 502437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14940.753866] LNet: Removed LNI 192.168.204.145@tcp [14940.756835] LNet: Removed LNI 192.168.204.145@tcp10 [14940.760668] LNet: Removed LNI 192.168.204.145@tcp11 [14940.774617] LNet: Removed LNI 192.168.204.145@tcp12 [14940.788387] LNet: Removed LNI 192.168.204.145@tcp13 [14940.796491] LNet: Removed LNI 192.168.204.145@tcp14 [14940.800194] LNet: Removed LNI 192.168.204.145@tcp15 [14940.807752] LNet: Removed LNI 192.168.204.145@tcp16 [14940.812873] LNet: Removed LNI 192.168.204.145@tcp17 [14940.821390] LNet: Removed LNI 192.168.204.145@tcp18 [14940.830153] LNet: Removed LNI 192.168.204.145@tcp19 [14940.834724] LNet: Removed LNI 192.168.204.145@tcp20 [14940.839663] LNet: Removed LNI 192.168.204.145@tcp21 [14940.844171] LNet: Removed LNI 192.168.204.145@tcp22 [14940.851818] LNet: Removed LNI 192.168.204.145@tcp23 [14940.860329] LNet: Removed LNI 192.168.204.145@tcp24 [14940.866233] LNet: Removed LNI 192.168.204.145@tcp25 [14940.870562] LNet: Removed LNI 192.168.204.145@tcp26 [14940.874485] LNet: Removed LNI 192.168.204.145@tcp27 [14940.877570] LNet: Removed LNI 192.168.204.145@tcp28 [14940.887256] LNet: Removed LNI 192.168.204.145@tcp29 [14940.890382] LNet: Removed LNI 192.168.204.145@tcp30 [14940.893352] LNet: Removed LNI 192.168.204.145@tcp31 [14940.903710] LNet: Removed LNI 192.168.204.145@tcp32 [14940.905734] LNet: Removed LNI 192.168.204.145@tcp33 [14940.915215] LNet: Removed LNI 192.168.204.145@tcp34 [14940.918379] LNet: Removed LNI 192.168.204.145@tcp35 [14940.922408] LNet: Removed LNI 192.168.204.145@tcp36 [14940.926667] LNet: Removed LNI 192.168.204.145@tcp37 [14940.930386] LNet: Removed LNI 192.168.204.145@tcp38 [14940.934342] LNet: Removed LNI 192.168.204.145@tcp39 [14940.938147] LNet: Removed LNI 192.168.204.145@tcp40 [14940.942334] LNet: Removed LNI 192.168.204.145@tcp41 [14940.946658] LNet: Removed LNI 192.168.204.145@tcp42 [14940.950789] LNet: Removed LNI 192.168.204.145@tcp43 [14940.954790] LNet: Removed LNI 192.168.204.145@tcp44 [14940.958549] LNet: Removed LNI 192.168.204.145@tcp45 [14940.961620] LNet: Removed LNI 192.168.204.145@tcp46 [14940.965480] LNet: Removed LNI 192.168.204.145@tcp47 [14940.970857] LNet: Removed LNI 192.168.204.145@tcp48 [14940.976799] LNet: Removed LNI 192.168.204.145@tcp49 [14941.001379] LNet: Removed LNI 192.168.204.145@tcp50 [14941.985364] Key type .llcrypt unregistered [14941.988077] Key type ._llcrypt unregistered [14961.385914] Key type ._llcrypt registered [14961.388517] Key type .llcrypt registered [14961.462633] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [14976.261379] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [14977.375857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14977.531681] alg: No test for adler32 (adler32-zlib) [14978.563756] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [14978.837476] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [14980.520388] Key type lgssc registered [14981.537498] Lustre: Echo OBD driver; http://www.lustre.org/ [15010.877049] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15010.912177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15012.149374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15012.191767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15012.306993] Lustre: lustre-MDT0000: new disk, initializing [15012.415458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15012.434717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15016.545361] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15026.865894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15027.009428] Lustre: 505638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15027.080387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15027.084386] Lustre: Skipped 1 previous similar message [15027.208218] Lustre: lustre-MDT0001: new disk, initializing [15027.332415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15027.377875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15027.389627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15032.424681] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15041.399755] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15047.236900] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15056.011368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15056.225658] Lustre: *** cfs_fail_loc=90f, val=0*** [15056.312152] Lustre: lustre-OST0000: new disk, initializing [15056.315688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15056.319936] Lustre: 507655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15056.392723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15063.386752] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15066.747428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15066.754201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15066.883374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15072.158326] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15082.467941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15082.476151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15082.482148] LustreError: Skipped 1 previous similar message [15082.508028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15084.947360] Lustre: server umount lustre-OST0000 complete [15089.122327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15089.131424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15089.157183] Lustre: Skipped 1 previous similar message [15089.162169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15089.166544] Lustre: Skipped 1 previous similar message [15092.707300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15092.708950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15094.674517] Lustre: server umount lustre-MDT0000 complete [15097.898282] LustreError: 505629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788324695 with bad export cookie 14359381374587488495 [15097.898938] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15097.903847] LustreError: 505629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15098.221423] Lustre: server umount lustre-MDT0001 complete [15108.522685] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15111.361349] Key type lgssc unregistered [15111.672768] LNet: 509561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15111.682419] LNetError: 509561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15111.699123] LNet: Removed LNI 192.168.204.145@tcp [15112.525750] Key type .llcrypt unregistered [15112.527793] Key type ._llcrypt unregistered [15130.748878] Key type ._llcrypt registered [15130.750586] Key type .llcrypt registered [15130.855662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15131.967691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15132.049628] alg: No test for adler32 (adler32-zlib) [15133.193740] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [15133.430344] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15135.136248] Key type lgssc registered [15136.806061] Lustre: Echo OBD driver; http://www.lustre.org/ [15144.674361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15144.688579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15146.132583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15150.587619] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15156.706687] LustreError: 510834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15156.741255] LustreError: 510834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15160.079332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15160.394374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15165.430791] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15174.833222] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15180.396168] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15187.843129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15188.254016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15194.901248] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15201.805611] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15214.661988] loop0: detected capacity change from 0 to 204800000 [15214.884693] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15214.895681] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15214.903791] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15220.892454] loop0: detected capacity change from 0 to 204800000 [15221.089456] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15221.109202] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15221.135845] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15226.699070] loop0: detected capacity change from 0 to 204800000 [15232.842839] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15233.002481] Lustre: fs15246-OST0000: new disk, initializing [15233.006677] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15233.013733] Lustre: 514431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15233.075161] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15239.861242] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15250.070103] loop1: detected capacity change from 0 to 204800000 [15255.927892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15256.166805] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15256.216168] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15256.289117] Lustre: fs15246-MDT0000: new disk, initializing [15256.488743] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15256.514776] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15256.614038] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15257.644114] Lustre: 515555:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000130f0126 x1875194392131712/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788324870 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15261.544384] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15266.732320] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15266.756915] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15266.770133] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15285.734149] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15285.748758] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15289.427421] Lustre: server umount fs15246-MDT0000 complete [15290.854224] LustreError: 510829:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15295.976799] LustreError: 510829:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15297.432375] Lustre: server umount fs15246-OST0000 complete [15311.330516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15311.351754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15311.365588] Lustre: Skipped 1 previous similar message [15311.380601] Lustre: Skipped 1 previous similar message [15315.884402] Lustre: server umount lustre-OST0000 complete [15321.570763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15321.584179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15321.604890] Lustre: Skipped 1 previous similar message [15321.618949] Lustre: Skipped 1 previous similar message [15325.895391] Lustre: server umount lustre-MDT0000 complete [15326.694838] LustreError: 510834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15326.721886] LustreError: 510834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15330.152638] LustreError: 515549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788324927 with bad export cookie 3202493821914988578 [15330.157298] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15330.171701] LustreError: 515549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15330.824527] Lustre: server umount lustre-MDT0001 complete [15342.812745] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15346.215471] Key type lgssc unregistered [15346.550781] LNet: 518654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15346.559954] LNetError: 518654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15346.573990] LNet: Removed LNI 192.168.204.145@tcp [15347.479722] Key type .llcrypt unregistered [15347.482623] Key type ._llcrypt unregistered [15367.021932] Key type ._llcrypt registered [15367.023563] Key type .llcrypt registered [15367.173837] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15368.220721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15368.247126] alg: No test for adler32 (adler32-zlib) [15369.503663] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [15369.864839] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [15371.538670] Key type lgssc registered [15372.888912] Lustre: Echo OBD driver; http://www.lustre.org/ [15413.689260] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15425.593321] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15463.400757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15463.439110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15464.760272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15464.802953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15464.922722] Lustre: lustre-MDT0000: new disk, initializing [15465.003381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15465.031753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15469.433488] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15481.119691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15481.250903] Lustre: 523122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15481.309718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15481.310721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15481.313414] Lustre: Skipped 1 previous similar message [15481.363487] Lustre: lustre-MDT0001: new disk, initializing [15481.435409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15481.498615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15481.513219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15485.974274] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15495.826542] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15502.656410] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15510.785649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15511.269670] Lustre: lustre-OST0000: new disk, initializing [15511.279675] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15511.298713] Lustre: 525092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15511.455167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15518.020130] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [15520.822943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15520.846063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15521.046370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15527.764683] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15533.710835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15534.127990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15538.360171] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15538.523972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15541.732169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15541.743580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15541.756923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15541.759447] Lustre: Skipped 1 previous similar message [15546.598427] Lustre: server umount lustre-OST0000 complete [15551.969556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15551.985307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15552.010035] Lustre: Skipped 2 previous similar messages [15552.023887] Lustre: Skipped 1 previous similar message [15557.094511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15557.106020] Lustre: Skipped 1 previous similar message [15559.790035] Lustre: server umount lustre-MDT0000 complete [15563.997347] LustreError: 523114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788325161 with bad export cookie 1100388798130871412 [15563.997907] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15564.012795] LustreError: 523114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15564.627174] Lustre: server umount lustre-MDT0001 complete [15595.532237] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [15606.602968] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [15643.050404] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15350 sec ========= 01:00:38 (1788325238) [15645.621570] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:00:40 (1788325240) === [15650.315187] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:00:45 (1788325245) === [15692.921506] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [15695.807274] Key type lgssc unregistered [15696.097304] LNet: 531755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15696.103811] LNetError: 531755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15696.118291] LNet: Removed LNI 192.168.204.145@tcp [15697.019224] Key type .llcrypt unregistered [15697.021421] Key type ._llcrypt unregistered