[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527154909 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002396] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007401] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010151] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013083] *** VALIDATE selinux *** [ 0.022542] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027758] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029164] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.032066] *** VALIDATE proc *** [ 0.033325] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035328] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036253] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039016] Speculative Store Bypass: Vulnerable [ 0.042534] debug: unmapping init [mem 0xffffffff84c59000-0xffffffff84c60fff] [ 0.045197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046773] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047036] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054362] rcu: Hierarchical SRCU implementation. [ 0.056833] smp: Bringing up secondary CPUs ... [ 0.057697] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.062103] smp: Brought up 1 node, 4 CPUs [ 0.064033] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.252780] node 0 deferred pages initialised in 184ms [ 0.256227] devtmpfs: initialized [ 0.257275] x86/mm: Memory block size: 128MB [ 0.259710] gcov: version magic: 0x41383552 [ 0.261185] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.262088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.263408] pinctrl core: initialized pinctrl subsystem [ 0.264225] [ 0.264905] ************************************************************* [ 0.265016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266015] ** ** [ 0.267016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268019] ** ** [ 0.269015] ** This means that this kernel is built to expose internal ** [ 0.270024] ** IOMMU data structures, which may compromise security on ** [ 0.271015] ** your system. ** [ 0.272016] ** ** [ 0.273015] ** If you see this message and you are not debugging the ** [ 0.274014] ** kernel, report this immediately to your vendor! ** [ 0.275017] ** ** [ 0.276019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.277015] ************************************************************* [ 0.278760] NET: Registered protocol family 16 [ 0.279517] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.280062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.281081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.282521] cpuidle: using governor menu [ 0.284959] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.287752] PCI: Using configuration type 1 for base access [ 0.289166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.299065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.300031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.303042] cryptd: max_cpu_qlen set to 1000 [ 0.305313] ACPI: Added _OSI(Module Device) [ 0.306029] ACPI: Added _OSI(Processor Device) [ 0.307040] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312329] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315484] ACPI: Interpreter enabled [ 0.317095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319025] ACPI: Using IOAPIC for interrupt routing [ 0.321169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.324484] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.335903] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338146] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.341031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.345142] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.350794] acpiphp: Slot [2] registered [ 0.353225] acpiphp: Slot [5] registered [ 0.354235] acpiphp: Slot [6] registered [ 0.356176] acpiphp: Slot [7] registered [ 0.358195] acpiphp: Slot [8] registered [ 0.359259] acpiphp: Slot [9] registered [ 0.361173] acpiphp: Slot [10] registered [ 0.363171] acpiphp: Slot [3] registered [ 0.365144] acpiphp: Slot [4] registered [ 0.366151] acpiphp: Slot [11] registered [ 0.368148] acpiphp: Slot [12] registered [ 0.370207] acpiphp: Slot [13] registered [ 0.372156] acpiphp: Slot [14] registered [ 0.373233] acpiphp: Slot [15] registered [ 0.375263] acpiphp: Slot [16] registered [ 0.377175] acpiphp: Slot [17] registered [ 0.378198] acpiphp: Slot [18] registered [ 0.379217] acpiphp: Slot [19] registered [ 0.381154] acpiphp: Slot [20] registered [ 0.383223] acpiphp: Slot [21] registered [ 0.385207] acpiphp: Slot [22] registered [ 0.386158] acpiphp: Slot [23] registered [ 0.388191] acpiphp: Slot [24] registered [ 0.390154] acpiphp: Slot [25] registered [ 0.391240] acpiphp: Slot [26] registered [ 0.393171] acpiphp: Slot [27] registered [ 0.395188] acpiphp: Slot [28] registered [ 0.396150] acpiphp: Slot [29] registered [ 0.398218] acpiphp: Slot [30] registered [ 0.400208] acpiphp: Slot [31] registered [ 0.401169] PCI host bridge to bus 0000:00 [ 0.403023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.405035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.408027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.410029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.413026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.416037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.418252] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.421222] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.425642] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435792] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.441523] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.444033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.445039] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449056] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.452489] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.455853] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.458898] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.462876] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.468018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.484025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.488964] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.495584] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.506018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.517021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.534019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543296] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.552025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.560021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.580020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.591567] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.600023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.608026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.630018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.643517] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.659031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.668014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.684018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.693486] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.704019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.712079] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.730026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740039] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.746015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.751022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.768020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.785390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.787338] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.790477] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.792303] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.798048] iommu: Default domain type: Passthrough [ 0.800556] SCSI subsystem initialized [ 0.802235] ACPI: bus type USB registered [ 0.803108] usbcore: registered new interface driver usbfs [ 0.805118] usbcore: registered new interface driver hub [ 0.807104] usbcore: registered new device driver usb [ 0.809285] pps_core: LinuxPPS API ver. 1 registered [ 0.811018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.814083] PTP clock support registered [ 0.816110] EDAC MC: Ver: 3.0.0 [ 0.817515] PCI: Using ACPI for IRQ routing [ 0.818632] NetLabel: Initializing [ 0.819011] NetLabel: domain hash size = 128 [ 0.820014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.821247] NetLabel: unlabeled traffic allowed by default [ 0.824067] vgaarb: loaded [ 0.825247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.827016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.838000] clocksource: Switched to clocksource kvm-clock [ 0.949917] VFS: Disk quotas dquot_6.6.0 [ 0.951604] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954841] *** VALIDATE ramfs *** [ 0.956166] *** VALIDATE hugetlbfs *** [ 0.957694] pnp: PnP ACPI init [ 0.960037] pnp: PnP ACPI: found 6 devices [ 0.978707] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.982779] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.985503] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.988027] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.990679] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.993089] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.996230] NET: Registered protocol family 2 [ 0.998678] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.003684] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.007687] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.013372] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.017161] TCP: Hash tables configured (established 65536 bind 65536) [ 1.020370] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.023611] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026498] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029987] NET: Registered protocol family 1 [ 1.032768] RPC: Registered named UNIX socket transport module. [ 1.035341] RPC: Registered udp transport module. [ 1.037156] RPC: Registered tcp transport module. [ 1.038814] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041291] NET: Registered protocol family 44 [ 1.043374] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.045768] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048194] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.050700] PCI: CLS 0 bytes, default 64 [ 1.053095] Unpacking initramfs... [ 2.496830] debug: unmapping init [mem 0xffff91ed7cc54000-0xffff91ed7ffbffff] [ 2.501225] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.503810] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.507147] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.138842] Initialise system trusted keyrings [ 3.140451] Key type blacklist registered [ 3.143928] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.152933] zbud: loaded [ 3.155902] *** VALIDATE nfs *** [ 3.157055] *** VALIDATE nfs4 *** [ 3.158473] pstore: using deflate compression [ 3.161643] Platform Keyring initialized [ 3.294437] NET: Registered protocol family 38 [ 3.296226] Key type asymmetric registered [ 3.297419] Asymmetric key parser 'x509' registered [ 3.299184] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.302556] io scheduler mq-deadline registered [ 3.304233] io scheduler kyber registered [ 3.305712] io scheduler bfq registered [ 3.309561] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.312533] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.315692] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.318636] ACPI: Power Button [PWRF] [ 3.324713] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.331962] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.349379] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.358177] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.376675] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.405874] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.435703] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.441694] Non-volatile memory driver v1.3 [ 3.443890] Linux agpgart interface v0.103 [ 3.483073] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.486351] vda: detected capacity change from 0 to 74711040 [ 3.505873] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.509553] vdb: detected capacity change from 0 to 1073741824 [ 3.528282] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.531553] vdc: detected capacity change from 0 to 2621440000 [ 3.547214] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.550745] vdd: detected capacity change from 0 to 2621440000 [ 3.568364] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.572712] vde: detected capacity change from 0 to 4294967296 [ 3.593777] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.597059] vdf: detected capacity change from 0 to 4294967296 [ 3.605911] libphy: Fixed MDIO Bus: probed [ 3.610990] usbcore: registered new interface driver usbserial_generic [ 3.613621] usbserial: USB Serial support registered for generic [ 3.615910] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.619842] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.621700] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.623946] mousedev: PS/2 mouse device common for all mice [ 3.629917] rtc_cmos 00:05: RTC can wake from S4 [ 3.632492] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.633985] rtc_cmos 00:05: registered as rtc0 [ 3.641527] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.641592] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.646309] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.650201] intel_pstate: CPU model not supported [ 3.655808] hid: raw HID events driver (C) Jiri Kosina [ 3.658039] usbcore: registered new interface driver usbhid [ 3.660409] usbhid: USB HID core driver [ 3.661958] drop_monitor: Initializing network drop monitor service [ 3.664605] Initializing XFRM netlink socket [ 3.666677] NET: Registered protocol family 10 [ 3.669528] Segment Routing with IPv6 [ 3.671312] NET: Registered protocol family 17 [ 3.674365] mpls_gso: MPLS GSO support [ 3.681159] RAS: Correctable Errors collector initialized. [ 3.682960] AVX version of gcm_enc/dec engaged. [ 3.684466] AES CTR mode by8 optimization enabled [ 3.794269] sched_clock: Marking stable (3794231427, 0)->(4815821768, -1021590341) [ 3.798694] registered taskstats version 1 [ 3.800958] Loading compiled-in X.509 certificates [ 3.803270] zswap: loaded using pool lzo/zbud [ 3.833878] Key type big_key registered [ 3.848269] Key type encrypted registered [ 3.850157] ima: No TPM chip found, activating TPM-bypass! [ 3.852466] ima: Allocated hash algorithm: sha1 [ 3.854314] ima: No architecture policies found [ 3.856307] evm: Initialising EVM extended attributes: [ 3.858374] evm: security.selinux [ 3.859462] evm: security.ima [ 3.860494] evm: security.capability [ 3.861638] evm: HMAC attrs: 0x1 [ 3.864125] rtc_cmos 00:05: setting system clock to 2026-08-17 06:14:36 UTC (1786947276) [ 3.871181] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 3.874644] debug: unmapping init [mem 0xffffffff84982000-0xffffffff84c58fff] [ 3.885185] Write protecting the kernel read-only data: 28672k [ 3.888881] debug: unmapping init [mem 0xffffffff83003000-0xffffffff831fffff] [ 3.892104] debug: unmapping init [mem 0xffffffff83914000-0xffffffff839fffff] [ 3.936600] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.946091] systemd[1]: Detected virtualization kvm. [ 3.948410] systemd[1]: Detected architecture x86-64. [ 3.950651] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.975532] systemd[1]: No hostname configured. [ 3.977723] systemd[1]: Set hostname to . [ 3.980033] random: systemd: uninitialized urandom read (16 bytes read) [ 3.982884] systemd[1]: Initializing machine ID from random generator. [ 4.024566] random: ln: uninitialized urandom read (6 bytes read) [ 4.135650] random: systemd: uninitialized urandom read (16 bytes read) [ 4.138487] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.144272] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.151282] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.873134] device-mapper: uevent: version 1.0.3 [ 4.875500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.659766] virtio_net virtio0 ens2: renamed from eth0 [ 5.673604] random: fast init done [ 5.735235] scsi host0: ata_piix [ 5.751707] scsi host1: ata_piix [ 5.755304] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.758403] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.113330] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.369213] random: crng init done [ 10.370658] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.893471] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.078230] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.341874] SELinux: Disabled at runtime. [ 12.401794] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.409706] systemd[1]: Detected virtualization kvm. [ 12.411396] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.004346] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.009052] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.014277] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.019088] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.022277] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.030163] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.035933] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ 13.193246] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.484968] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.821505] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.875413] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.023622] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.033539] EDAC sbridge: Ver: 1.1.2 [ 15.751836] Key type dns_resolver registered [ 16.067048] NFS: Registering the id_resolver key type [ 16.069585] Key type id_resolver registered [ 16.071373] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg405-server login: [ 48.956094] libcfs: loading out-of-tree module taints kernel. [ 49.045659] Key type ._llcrypt registered [ 49.055216] Key type .llcrypt registered [ 49.226344] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 71.361575] hrtimer: interrupt took 4563774 ns [ 72.794938] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 75.585099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.631540] alg: No test for adler32 (adler32-zlib) [ 77.554370] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 79.048321] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 81.024667] Key type lgssc registered [ 83.397965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.812818] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.476576] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 192.179559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.244806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.623900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.669518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.856087] Lustre: lustre-MDT0000: new disk, initializing [ 194.007894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.068083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.889340] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.320326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.511444] Lustre: 6532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 221.558520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 221.570056] Lustre: Skipped 1 previous similar message [ 221.734298] Lustre: lustre-MDT0001: new disk, initializing [ 221.838077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.898063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 221.919640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.368191] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.484952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.557337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.977719] Lustre: lustre-OST0000: new disk, initializing [ 247.982167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 247.987748] Lustre: 8474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.130861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.039712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 254.057246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 254.165254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 256.847715] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.226418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.466162] Lustre: lustre-OST0001: new disk, initializing [ 277.472425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.481878] Lustre: 9547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.625186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 286.412945] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.308916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.334544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.505530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 301.509067] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.856171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.643837] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing check_logdir /tmp/testlogs/ [ 327.209794] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing yml_node [ 332.762806] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 336.875220] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 340.302939] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 342.534395] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 17 02:20:13 EDT 2026 [ 364.315671] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 366.498570] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 385.006222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.011868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.036499] Lustre: Skipped 1 previous similar message [ 385.093245] Lustre: Skipped 3 previous similar messages [ 390.116402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.124589] Lustre: Skipped 3 previous similar messages [ 395.239100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 398.303229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.584975] Lustre: server umount lustre-MDT0000 complete [ 403.413329] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947676 with bad export cookie 13702719099699941872 [ 403.414302] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.419416] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 405.472505] LustreError: 12294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.473300] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.474027] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 405.474033] Lustre: Skipped 3 previous similar messages [ 405.513304] LustreError: 12294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 405.532610] Lustre: Skipped 3 previous similar messages [ 408.480908] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.515761] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 410.036444] Lustre: server umount lustre-MDT0001 complete [ 415.475594] Lustre: server umount lustre-OST0000 complete [ 420.491968] Lustre: server umount lustre-OST0001 complete [ 429.899337] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 442.779201] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 488.636452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.055727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 489.110423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 489.209709] Lustre: lustre-MDT0000: new disk, initializing [ 489.339726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.371691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 495.105865] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 508.639671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.778488] Lustre: 15106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 508.872784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 508.895250] Lustre: Skipped 1 previous similar message [ 509.063245] Lustre: lustre-MDT0001: new disk, initializing [ 509.298646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 509.390094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 509.406514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 514.636088] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 527.716580] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 535.311967] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 544.518082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.035301] Lustre: lustre-OST0000: new disk, initializing [ 545.042925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 545.056738] Lustre: 17080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 545.267696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.642110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 546.669080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 546.829805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 552.973678] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 565.112571] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 572.518865] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 572.873642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 579.433245] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 579.628599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 582.626993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.665777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.680734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 582.695821] Lustre: Skipped 4 previous similar messages [ 588.444828] Lustre: server umount lustre-OST0000 complete [ 596.450725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 596.461504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.479662] Lustre: Skipped 1 previous similar message [ 596.490451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.496487] Lustre: Skipped 2 previous similar messages [ 599.742191] Lustre: server umount lustre-MDT0000 complete [ 603.105761] LustreError: 15113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.121501] LustreError: 15113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 603.905284] LustreError: 15098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786947876 with bad export cookie 13702719099699943405 [ 603.922973] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.940763] LustreError: 15098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 604.342871] Lustre: server umount lustre-MDT0001 complete [ 609.225457] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:24:39 (1786947879) [ 612.657679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 615.087420] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:24:45 (1786947885) [ 624.956418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.663063] LustreError: 19491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.833410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.267068] LustreError: 19492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.952439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 641.512203] LustreError: 19491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.529291] LustreError: 19491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 643.532241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.417635] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 658.604416] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 665.194629] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 674.158221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.622933] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 692.276328] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 709.600394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 709.617671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.636493] Lustre: Skipped 1 previous similar message [ 709.642711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 709.649763] Lustre: Skipped 2 previous similar messages [ 712.685929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 714.052852] Lustre: server umount lustre-OST0000 complete [ 721.377638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 721.393841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.408792] Lustre: Skipped 1 previous similar message [ 733.151929] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 733.523551] Lustre: server umount lustre-MDT0000 complete [ 738.169608] LustreError: 19472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948010 with bad export cookie 13702719099699944287 [ 738.177873] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 738.188105] LustreError: 19472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 738.756663] Lustre: server umount lustre-MDT0001 complete [ 754.597467] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 757.974553] Key type lgssc unregistered [ 758.508535] LNet: 23055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.522729] LNetError: 23055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.541445] LNet: Removed LNI 192.168.204.105@tcp [ 759.748179] Key type .llcrypt unregistered [ 759.750247] Key type ._llcrypt unregistered [ 787.887414] Key type ._llcrypt registered [ 787.890924] Key type .llcrypt registered [ 788.010687] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 789.341470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.375578] alg: No test for adler32 (adler32-zlib) [ 790.529197] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 790.833530] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 792.527498] Key type lgssc registered [ 793.958356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 803.803073] loop: module loaded [ 803.887869] loop0: detected capacity change from 0 to 204800000 [ 804.198193] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 804.204599] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 804.676820] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 811.934746] loop0: detected capacity change from 0 to 204800000 [ 812.189925] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 812.218551] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 812.253751] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 819.096178] loop0: detected capacity change from 0 to 204800000 [ 825.899574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 825.942093] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.213130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 827.296214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 827.421596] Lustre: lustre-MDT0000: new disk, initializing [ 827.541133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.564886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 833.268164] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 844.011558] loop1: detected capacity change from 0 to 204800000 [ 850.798162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 851.132651] Lustre: lustre-OST07c6: new disk, initializing [ 851.136593] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 851.144775] Lustre: Skipped 1 previous similar message [ 851.155183] Lustre: 26243:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 851.242958] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 857.307025] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 860.777330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 860.784824] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 860.826235] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 862.930695] Lustre: 25230:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 862.949410] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 862.963353] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 862.972114] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 862.987666] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 862.995480] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 880.201442] Lustre: 25229:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 880.211352] Lustre: 25229:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 880.221198] Lustre: 25229:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 880.228506] Lustre: 25229:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 880.237630] Lustre: 25229:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 880.243533] Lustre: 25229:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 885.645072] Lustre: 25230:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 885.653131] Lustre: 25230:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 885.659299] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 885.664880] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.669546] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 885.674583] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.680323] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 885.688097] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.693546] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 885.699643] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 885.705433] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 885.712981] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 890.618725] Lustre: 25231:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 890.634787] Lustre: 25231:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 890.642212] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 890.651087] Lustre: 25231:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 890.660081] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 890.666681] Lustre: 25231:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 890.676686] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 890.691914] Lustre: 25231:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 890.698048] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 890.702761] Lustre: 25231:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 890.708150] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 890.713410] Lustre: 25231:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 899.017373] Lustre: 25229:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 899.024238] Lustre: 25229:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 899.031426] Lustre: 25229:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 899.046799] Lustre: 25229:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 899.054032] Lustre: 25229:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 899.061365] Lustre: 25229:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 899.073806] Lustre: 25229:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 899.078716] Lustre: 25229:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 899.085780] Lustre: 25229:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 899.092026] Lustre: 25229:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 899.098908] Lustre: 25229:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 899.106170] Lustre: 25229:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 907.506679] Lustre: 25230:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 907.518928] Lustre: 25230:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 907.533375] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 907.546133] Lustre: 25230:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 907.562172] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 907.567690] Lustre: 25230:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 907.574390] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 907.583223] Lustre: 25230:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 907.592364] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 907.599905] Lustre: 25230:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 907.607710] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 907.619099] Lustre: 25230:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 957.921778] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.923417] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.927565] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 962.217276] Lustre: server umount lustre-OST07c6 complete [ 973.604351] Lustre: server umount lustre-MDT0000 complete [ 992.113812] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 995.009124] Key type lgssc unregistered [ 995.405853] LNet: 28806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.413798] LNetError: 28806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.425741] LNet: Removed LNI 192.168.204.105@tcp [ 996.257289] Key type .llcrypt unregistered [ 996.259368] Key type ._llcrypt unregistered [ 1016.639883] Key type ._llcrypt registered [ 1016.648480] Key type .llcrypt registered [ 1016.785746] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1017.951104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1018.009859] alg: No test for adler32 (adler32-zlib) [ 1019.286921] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1019.672183] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1021.423192] Key type lgssc registered [ 1022.783701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.624552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.653307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1033.346565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.695609] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1042.936205] LustreError: 30077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.960685] LustreError: 30077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1048.036666] LustreError: 30078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.339204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.969164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.870116] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1061.512576] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1069.269672] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.058028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.632182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.219625] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1096.671913] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.250603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 1118.693811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.699661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.704697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.718192] Lustre: Skipped 1 previous similar message [ 1118.754093] Lustre: Skipped 1 previous similar message [ 1123.183975] Lustre: server umount lustre-OST0000 complete [ 1128.949174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.959760] Lustre: Skipped 1 previous similar message [ 1128.960430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.976554] Lustre: Skipped 1 previous similar message [ 1133.780281] Lustre: server umount lustre-MDT0000 complete [ 1134.051652] LustreError: 30077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.084376] LustreError: 30077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1138.412693] LustreError: 30059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948411 with bad export cookie 13760320885746783392 [ 1138.419144] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.423885] LustreError: 30059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1138.715183] Lustre: server umount lustre-MDT0001 complete [ 1154.628870] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1158.756506] Key type lgssc unregistered [ 1159.083793] LNet: 33819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.108751] LNetError: 33819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.171962] LNet: Removed LNI 192.168.204.105@tcp [ 1160.770822] Key type .llcrypt unregistered [ 1160.778186] Key type ._llcrypt unregistered [ 1184.333994] Key type ._llcrypt registered [ 1184.336912] Key type .llcrypt registered [ 1184.452711] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1185.584836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1185.760844] alg: No test for adler32 (adler32-zlib) [ 1186.841305] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1187.062213] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1188.703984] Key type lgssc registered [ 1189.725599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1198.072659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1198.097990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.708193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1205.443879] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1209.828479] LustreError: 35090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.849407] LustreError: 35090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1214.947420] LustreError: 35091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.400286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.720559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.766628] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1229.434426] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1235.283704] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1242.531158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.802109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.864442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1249.221106] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1257.536536] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.874395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1275.881699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.895331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1276.897591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.901820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1276.902791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.352764] Lustre: server umount lustre-OST0000 complete [ 1287.143252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.149120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.157679] Lustre: Skipped 1 previous similar message [ 1287.179580] Lustre: Skipped 2 previous similar messages [ 1289.998176] Lustre: server umount lustre-MDT0000 complete [ 1292.260624] LustreError: 35821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.288891] LustreError: 35821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1294.729090] LustreError: 35071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948567 with bad export cookie 14913310650021509204 [ 1294.731720] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.742753] LustreError: 35071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1295.239885] Lustre: server umount lustre-MDT0001 complete [ 1311.029935] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1313.492606] Key type lgssc unregistered [ 1313.785375] LNet: 38650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.800095] LNetError: 38650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.822674] LNet: Removed LNI 192.168.204.105@tcp [ 1314.784505] Key type .llcrypt unregistered [ 1314.788461] Key type ._llcrypt unregistered [ 1342.033332] Key type ._llcrypt registered [ 1342.036491] Key type .llcrypt registered [ 1342.167424] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1343.332371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.408061] alg: No test for adler32 (adler32-zlib) [ 1344.516498] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1344.851404] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1346.551187] Key type lgssc registered [ 1347.944785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1356.499476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1356.514492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.219728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.881417] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1367.523596] LustreError: 40068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.564854] LustreError: 40068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1372.646739] LustreError: 40069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.811287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.229403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1378.020812] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1389.224804] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.941334] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1406.500286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1406.955209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.235717] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1424.080197] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1427.473617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1436.227899] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 1437.668031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.681384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.694689] Lustre: Skipped 1 previous similar message [ 1437.710059] Lustre: Skipped 2 previous similar messages [ 1438.418774] Lustre: server umount lustre-MDT0000 complete [ 1441.306801] LustreError: 40063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.946623] LustreError: 40050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948715 with bad export cookie 13517048725010797975 [ 1442.954293] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.965047] LustreError: 40050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1446.448941] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.478110] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 1446.485752] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1447.925328] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.941514] Lustre: Skipped 1 previous similar message [ 1449.416541] Lustre: server umount lustre-MDT0001 complete [ 1470.833245] Lustre: server umount lustre-OST0000 complete [ 1490.751237] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1493.600918] Key type lgssc unregistered [ 1494.080745] LNet: 43935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.095492] LNetError: 43935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.115142] LNet: Removed LNI 192.168.204.105@tcp [ 1495.234117] Key type .llcrypt unregistered [ 1495.237485] Key type ._llcrypt unregistered [ 1522.895482] Key type ._llcrypt registered [ 1522.908449] Key type .llcrypt registered [ 1523.059690] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1524.311786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1524.606311] alg: No test for adler32 (adler32-zlib) [ 1525.720751] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1526.113563] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1527.895171] Key type lgssc registered [ 1529.450309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1539.194192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1539.229290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1540.769390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1546.299744] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1551.335922] LustreError: 45355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1551.360910] LustreError: 45355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1556.345224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.452508] LustreError: 45354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.749649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1562.269366] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1570.664131] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1577.975773] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1585.879770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1586.373441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.290131] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1602.117463] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1610.931737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1621.481577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.494397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1621.513954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.533952] Lustre: Skipped 1 previous similar message [ 1623.619463] Lustre: lustre-OST0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 1624.882295] Lustre: server umount lustre-OST0000 complete [ 1644.001149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1644.011202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.025380] Lustre: Skipped 1 previous similar message [ 1644.037139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.074150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.076505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.703470] Lustre: server umount lustre-MDT0000 complete [ 1652.192419] LustreError: 47719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.205032] LustreError: 47719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1653.861524] LustreError: 45336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786948926 with bad export cookie 5696574265747690004 [ 1653.863822] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1653.870680] LustreError: 45336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1660.180355] Lustre: server umount lustre-MDT0001 complete [ 1671.118778] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1674.432679] Key type lgssc unregistered [ 1674.850350] LNet: 49088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.868256] LNetError: 49088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.900396] LNet: Removed LNI 192.168.204.105@tcp [ 1675.901885] Key type .llcrypt unregistered [ 1675.907559] Key type ._llcrypt unregistered [ 1708.862690] Key type ._llcrypt registered [ 1708.867120] Key type .llcrypt registered [ 1709.106125] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1710.499359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.587737] alg: No test for adler32 (adler32-zlib) [ 1711.826098] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1712.277035] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1714.119315] Key type lgssc registered [ 1715.325487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.411576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.435658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.753171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.961380] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1737.192261] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.216767] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1742.311473] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.210255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.800936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1749.183152] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1758.076831] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1763.991972] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1771.573629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1771.881283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.343428] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1787.185898] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1796.510833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1803.842520] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1808.518764] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1821.863827] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-17 2h44m51s [ 1823.878823] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1827.302287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.307504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.331239] Lustre: Skipped 1 previous similar message [ 1827.342210] Lustre: Skipped 2 previous similar messages [ 1828.391079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1828.406723] Lustre: Skipped 2 previous similar messages [ 1829.975159] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 1829.987604] Lustre: Skipped 1 previous similar message [ 1832.542854] Lustre: server umount lustre-MDT0000 complete [ 1834.470353] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.483139] LustreError: 50506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1836.701182] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1840.613150] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1840.646235] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1844.845623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.002727] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.322290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.417191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1846.398789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1850.645643] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1855.049927] Lustre: DEBUG MARKER: Wait for df (36722) ... [ 1857.474727] Lustre: DEBUG MARKER: done [ 1865.698815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.706644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.723602] Lustre: Skipped 1 previous similar message [ 1865.729496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.212860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1869.576494] Lustre: server umount lustre-OST0000 complete [ 1877.501798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1877.501977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.507160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.507171] Lustre: Skipped 3 previous similar messages [ 1877.564396] Lustre: Skipped 2 previous similar messages [ 1881.505788] Lustre: server umount lustre-MDT0000 complete [ 1885.166672] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.196293] LustreError: 50507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1886.131722] LustreError: 50488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949158 with bad export cookie 3980596149389213791 [ 1886.149866] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1886.151428] LustreError: 50488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1886.725946] Lustre: server umount lustre-MDT0001 complete [ 1900.229400] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1903.771566] Key type lgssc unregistered [ 1904.160880] LNet: 55456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.175244] LNetError: 55456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.199790] LNet: Removed LNI 192.168.204.105@tcp [ 1905.294155] Key type .llcrypt unregistered [ 1905.298423] Key type ._llcrypt unregistered [ 1953.148163] Key type ._llcrypt registered [ 1953.154502] Key type .llcrypt registered [ 1953.316741] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1955.100935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.178758] alg: No test for adler32 (adler32-zlib) [ 1956.322769] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 1956.610543] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1958.287210] Key type lgssc registered [ 1959.722955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1968.853881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1968.877651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.213602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1970.256962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1970.480524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.641652] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1985.781367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.931438] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1985.988037] Lustre: 57682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1986.324551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.637191] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2001.286560] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2008.520294] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.007869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2017.307211] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2017.602149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.756692] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2026.224986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2034.667585] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2041.554575] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-17 2h48m32s [ 2043.584304] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2052.477840] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2055.799373] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2056.075340] LustreError: 57689:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2076.242617] Lustre: lustre-MDT0000: Client 37b39592-d9ee-499c-aa3a-d9ebbad67309 (at 192.168.204.5@tcp) reconnecting [ 2096.122765] LustreError: 57689:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2098.812614] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2101.975086] Lustre: DEBUG MARKER: done [ 2113.508428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.511844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.512152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.528305] Lustre: Skipped 1 previous similar message [ 2116.588603] Lustre: server umount lustre-OST0000 complete [ 2123.745098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.764194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.766802] Lustre: Skipped 1 previous similar message [ 2124.834966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.841032] Lustre: Skipped 1 previous similar message [ 2127.505613] Lustre: server umount lustre-MDT0000 complete [ 2130.922540] LustreError: 57692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.945334] LustreError: 57692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2131.637375] LustreError: 57672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949404 with bad export cookie 8276902579613466997 [ 2131.649729] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.653394] LustreError: 57672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2132.059495] Lustre: server umount lustre-MDT0001 complete [ 2146.910375] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2150.308485] Key type lgssc unregistered [ 2150.717540] LNet: 62017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.730376] LNetError: 62017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.760678] LNet: Removed LNI 192.168.204.105@tcp [ 2151.779423] Key type .llcrypt unregistered [ 2151.783688] Key type ._llcrypt unregistered [ 2204.878349] Key type ._llcrypt registered [ 2204.881105] Key type .llcrypt registered [ 2205.046835] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2206.567668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2206.604255] alg: No test for adler32 (adler32-zlib) [ 2207.677958] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2208.062293] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2209.791159] Key type lgssc registered [ 2211.216795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.869384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2214.894251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.256789] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2216.279926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2216.474566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.389171] Lustre: Failing over lustre-MDT0000 [ 2217.653754] Lustre: server umount lustre-MDT0000 complete [ 2225.075359] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:51:35 (1786949495) [ 2231.716536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.218759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.306586] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2245.830352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.964860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2246.009869] Lustre: 64821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2246.258853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2250.646706] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2259.420189] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2266.344596] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2273.661810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.880139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2274.082272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.670983] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2283.720670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 2288.981409] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2295.627725] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2304.481848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.488772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.511785] Lustre: Skipped 1 previous similar message [ 2304.532864] Lustre: Skipped 2 previous similar messages [ 2309.606733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.617841] Lustre: Skipped 2 previous similar messages [ 2310.127197] Lustre: server umount lustre-MDT0000 complete [ 2314.429442] LustreError: 64814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949587 with bad export cookie 1065646742622623823 [ 2314.441435] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.458347] LustreError: 64814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2314.731262] Lustre: server umount lustre-MDT0001 complete [ 2318.306102] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2323.547013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2331.105726] Lustre: 63745:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949587/real 1786949587] req@000000004480d9f1 x1873752337981824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2331.159022] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.172476] Lustre: Skipped 1 previous similar message [ 2335.152091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.223371] Lustre: 63744:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949592/real 1786949592] req@000000005d934c31 x1873752337981952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2340.394134] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec9f010fe12e132 [ 2340.432835] Lustre: MGC192.168.204.105@tcp: Connection restored to 0@lo (at 0@lo) [ 2340.941313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.090696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2341.932345] Lustre: 63746:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949597/real 1786949597] req@00000000f35ba6a3 x1873752337982080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2345.953204] LustreError: 66672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.970370] LustreError: 66672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2346.769737] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2346.975521] Lustre: 63746:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949602/real 1786949602] req@0000000015bd1149 x1873752337982336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2351.076485] LustreError: 66671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.195778] LustreError: 68593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.210937] LustreError: 68593:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2356.408547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.926040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2360.056187] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2362.634910] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2371.823278] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2379.343907] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2391.012415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.023627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.029908] Lustre: Skipped 2 previous similar messages [ 2391.046393] Lustre: Skipped 2 previous similar messages [ 2396.129934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.138859] Lustre: Skipped 2 previous similar messages [ 2401.254753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2401.267224] Lustre: Skipped 2 previous similar messages [ 2402.271702] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2402.525383] Lustre: server umount lustre-MDT0000 complete [ 2406.370324] LustreError: 69306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.391289] LustreError: 69306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2406.789181] LustreError: 64816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949679 with bad export cookie 1065646742622626098 [ 2406.797267] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.804311] LustreError: 64816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2407.295496] Lustre: server umount lustre-MDT0001 complete [ 2414.310332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2426.796906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.872136] Lustre: 63747:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949684/real 1786949684] req@00000000296f3182 x1873752338013184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2427.920664] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.876278] LustreError: 66671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.161638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.334193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2437.229708] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2437.240389] Lustre: Skipped 1 previous similar message [ 2438.152325] Lustre: 63745:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786949694/real 1786949694] req@000000009709c18a x1873752338013568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786949710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2438.170395] Lustre: 63745:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2439.585826] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2443.746713] LustreError: 71278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.765144] LustreError: 71278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2449.101071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.194632] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2455.046773] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2462.906458] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2469.714723] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2480.638265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.643843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.652692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.671395] Lustre: Skipped 1 previous similar message [ 2484.055290] Lustre: server umount lustre-MDT0000 complete [ 2485.735497] LustreError: 71278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.748749] LustreError: 71278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2488.419137] LustreError: 64814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949761 with bad export cookie 1065646742622627477 [ 2488.421784] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.428429] LustreError: 64814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2488.776972] Lustre: server umount lustre-MDT0001 complete [ 2495.770275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2499.603664] Lustre: DEBUG MARKER: files compared the same [ 2512.280301] Lustre: server umount lustre-OST0000 complete [ 2533.601122] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2536.768164] Key type lgssc unregistered [ 2537.029647] LNet: 74559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.033885] LNetError: 74559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.049292] LNet: Removed LNI 192.168.204.105@tcp [ 2537.837809] Key type .llcrypt unregistered [ 2537.841104] Key type ._llcrypt unregistered [ 2564.868700] Key type ._llcrypt registered [ 2564.870214] Key type .llcrypt registered [ 2565.026556] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2566.295852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2566.339156] alg: No test for adler32 (adler32-zlib) [ 2567.406029] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2567.726975] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2569.415184] Key type lgssc registered [ 2570.949380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.850988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2579.883413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.451200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.615975] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 2591.712677] LustreError: 75978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.744256] LustreError: 75978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2596.373248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.775208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2601.873978] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 2609.677180] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2615.232511] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2623.189412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2623.473044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.541719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2630.147165] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 2638.997200] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2652.131136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.138768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2652.148649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2652.162432] Lustre: Skipped 1 previous similar message [ 2657.255750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2657.260376] Lustre: Skipped 1 previous similar message [ 2658.036508] Lustre: server umount lustre-OST0000 complete [ 2662.372187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.381976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.387653] Lustre: Skipped 2 previous similar messages [ 2662.401904] Lustre: Skipped 1 previous similar message [ 2667.487878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.608382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.618338] Lustre: Skipped 2 previous similar messages [ 2676.191144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2676.330496] Lustre: server umount lustre-MDT0000 complete [ 2677.731741] LustreError: 75973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.761259] LustreError: 75973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2680.323925] LustreError: 75960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786949952 with bad export cookie 18099796860769020486 [ 2680.329517] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.345250] LustreError: 75960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2680.700812] Lustre: server umount lustre-MDT0001 complete [ 2692.520989] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2695.505786] Key type lgssc unregistered [ 2695.858721] LNet: 79538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.874930] LNetError: 79538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.897274] LNet: Removed LNI 192.168.204.105@tcp [ 2696.690559] Key type .llcrypt unregistered [ 2696.693918] Key type ._llcrypt unregistered [ 2722.825827] Key type ._llcrypt registered [ 2722.827770] Key type .llcrypt registered [ 2722.984480] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2724.731110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2724.777969] alg: No test for adler32 (adler32-zlib) [ 2725.976499] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2726.320363] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2728.076864] Key type lgssc registered [ 2729.371313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2738.680226] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2738.727941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.400617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2759.519717] LustreError: 80931:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [ 2786.207330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.127573] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2805.507525] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2814.983454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.127513] Lustre: *** cfs_fail_loc=706, val=0*** [ 2815.482335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.574558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2819.575410] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2820.593508] LustreError: 80957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.643426] LustreError: 80957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2825.713448] LustreError: 80956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.734698] LustreError: 80956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2829.542808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.031380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2834.668995] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2843.319402] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2851.012393] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2855.924396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.951572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.954857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.984059] Lustre: Skipped 1 previous similar message [ 2859.851382] Lustre: server umount lustre-OST0000 complete [ 2866.152334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.156560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2866.160691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2866.165685] Lustre: Skipped 2 previous similar messages [ 2869.912691] Lustre: server umount lustre-MDT0000 complete [ 2871.268521] LustreError: 83669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.300978] LustreError: 83669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2873.904334] LustreError: 82770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950146 with bad export cookie 16756922610424141312 [ 2873.906856] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2873.911076] LustreError: 82770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2874.127481] Lustre: server umount lustre-MDT0001 complete [ 2886.533977] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2889.686906] Key type lgssc unregistered [ 2889.992901] LNet: 84566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2890.003627] LNetError: 84566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.020819] LNet: Removed LNI 192.168.204.105@tcp [ 2891.106285] Key type .llcrypt unregistered [ 2891.108562] Key type ._llcrypt unregistered [ 2918.312713] Key type ._llcrypt registered [ 2918.316469] Key type .llcrypt registered [ 2918.424237] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2919.543185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2919.574072] alg: No test for adler32 (adler32-zlib) [ 2920.578853] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 2920.777922] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2922.455165] Key type lgssc registered [ 2924.466965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2933.884499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2933.902813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.430267] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2950.362647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.944817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2956.115480] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2956.258260] LustreError: 86673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.281410] LustreError: 86673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2961.379674] LustreError: 86672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.400405] LustreError: 86672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2966.499654] LustreError: 86677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.519528] LustreError: 86677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2967.506986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.854726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.158406] LustreError: 86672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.177213] LustreError: 86672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2975.683747] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2977.257319] LustreError: 86673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.287983] LustreError: 86673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2986.832475] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.846221] Lustre: MGS: Client 93b60c8f-0f06-4c31-829c-47ccce15610c (at 0@lo) reconnecting [ 2986.878738] LustreError: 86687:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2987.094238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.165805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2990.986758] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3004.389428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.408497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.430832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.465737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.468557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.496925] Lustre: Skipped 1 previous similar message [ 3009.819777] Lustre: server umount lustre-OST0000 complete [ 3017.700690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.703255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.726597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.736606] Lustre: Skipped 1 previous similar message [ 3020.196162] Lustre: server umount lustre-MDT0000 complete [ 3022.818434] LustreError: 88221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.834066] LustreError: 88221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3024.711447] Lustre: server umount lustre-MDT0001 complete [ 3029.050460] Lustre: server umount MGS complete [ 3041.865399] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:05:12 (1786950312) [ 3075.497085] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 3087.387385] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3132.743477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.320101] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3148.452535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.769577] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3148.823839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3148.880617] Lustre: lustre-MDT0001: new disk, initializing [ 3148.955661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3148.992986] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3148.999264] Lustre: 93844:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3149.009538] LustreError: 93844:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3153.985833] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3164.111421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3164.514974] Lustre: lustre-OST0000: new disk, initializing [ 3164.524837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3164.540443] Lustre: 94611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3164.633268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.805945] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3174.531633] Lustre: 94948:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000aed112a3 x1873753085617280/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786950463 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3185.929729] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3193.499728] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.508616] Lustre: MGS: Client f53a4051-a20e-48e4-b1a6-9d52d3258b41 (at 0@lo) reconnecting [ 3193.560923] Lustre: 95658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3193.581839] LustreError: 93830:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3193.609929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3193.682958] Lustre: lustre-MDT0000: new disk, initializing [ 3193.798316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.826754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3193.844542] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3194.919634] Lustre: 95683:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000478e9e32 x1873753085624448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786950483 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3198.794927] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3199.977126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3199.992487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3200.135541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3225.061101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.076239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.085249] Lustre: Skipped 1 previous similar message [ 3225.114617] Lustre: Skipped 2 previous similar messages [ 3230.178894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.198534] Lustre: Skipped 1 previous similar message [ 3231.090823] Lustre: server umount lustre-OST0000 complete [ 3235.327717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3235.335322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.364808] Lustre: Skipped 1 previous similar message [ 3240.418435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3240.425670] Lustre: Skipped 3 previous similar messages [ 3241.632782] Lustre: server umount lustre-MDT0000 complete [ 3245.537631] LustreError: 93814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.560957] LustreError: 93814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3260.383318] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3260.771185] Lustre: server umount lustre-MDT0001 complete [ 3265.329172] Lustre: server umount MGS complete [ 3282.861713] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:09:13 (1786950553) [ 3306.543437] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3309.767607] Key type lgssc unregistered [ 3310.176224] LNet: 98187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3310.185283] LNetError: 98187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3310.202276] LNet: Removed LNI 192.168.204.105@tcp [ 3311.175321] Key type .llcrypt unregistered [ 3311.182346] Key type ._llcrypt unregistered [ 3334.886420] Key type ._llcrypt registered [ 3334.890385] Key type .llcrypt registered [ 3335.156598] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3336.369034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3336.454376] alg: No test for adler32 (adler32-zlib) [ 3337.654872] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3338.060899] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3339.831383] Key type lgssc registered [ 3341.457590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3350.438560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3350.494342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.333852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.932756] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3361.765785] LustreError: 99412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.797558] LustreError: 99412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3366.882852] LustreError: 99413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.498243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.803801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.580861] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3380.545065] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3386.994675] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3394.385989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.786028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.860011] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3400.589812] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3408.872498] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3423.715750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.716980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.735350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.836885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.849926] Lustre: Skipped 2 previous similar messages [ 3429.491294] Lustre: server umount lustre-OST0000 complete [ 3433.953295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.962841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.976062] Lustre: Skipped 2 previous similar messages [ 3433.998035] Lustre: Skipped 1 previous similar message [ 3439.075088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.086258] Lustre: Skipped 1 previous similar message [ 3439.653752] Lustre: server umount lustre-MDT0000 complete [ 3444.192729] LustreError: 99412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.207362] LustreError: 99412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3444.580659] LustreError: 99393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950717 with bad export cookie 4608637330102360621 [ 3444.596637] LustreError: 99393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3444.598270] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.039019] Lustre: server umount lustre-MDT0001 complete [ 3456.667961] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3459.591395] Key type lgssc unregistered [ 3460.048920] LNet: 102981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.059495] LNetError: 102981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3460.094683] LNet: Removed LNI 192.168.204.105@tcp [ 3460.902843] Key type .llcrypt unregistered [ 3460.905565] Key type ._llcrypt unregistered [ 3478.978799] Key type ._llcrypt registered [ 3478.982414] Key type .llcrypt registered [ 3479.081302] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3480.026166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.098293] alg: No test for adler32 (adler32-zlib) [ 3481.321111] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3481.546342] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3483.191190] Key type lgssc registered [ 3484.703421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3491.890814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3491.901628] LustreError: 104073:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3491.913900] LustreError: 104074:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3491.925842] LustreError: 104073:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3491.932985] LustreError: 104073:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3491.941467] LustreError: 104073:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3491.964631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.569099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.610982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.007026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3506.890221] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3513.715414] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing lsmod [ 3517.756421] LustreError: 105280:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3517.819134] LustreError: 105281:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3517.833041] LustreError: 105280:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4938 [ 3517.843841] LustreError: 105280:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3517.874272] LustreError: 105280:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3517.887277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3518.500598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.559970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3522.529458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3522.547841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.581288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.580473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.583921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.620327] Lustre: Skipped 1 previous similar message [ 3524.647938] Lustre: Skipped 2 previous similar messages [ 3526.692379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.697736] Lustre: Skipped 1 previous similar message [ 3528.476776] Lustre: server umount lustre-MDT0000 complete [ 3531.747518] LustreError: 104112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.766570] LustreError: 104112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3532.718710] LustreError: 104089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950805 with bad export cookie 6758304886988018787 [ 3532.727070] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3532.741576] LustreError: 104089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3532.883816] LustreError: 104542:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006c954822 x1873753673848576/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3532.988301] Lustre: server umount lustre-MDT0001 complete [ 3547.492678] Lustre: server umount lustre-OST0000 complete [ 3556.637907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.130459] LustreError: 106345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.200803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.464983] LustreError: 106346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.206392] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3566.561250] LustreError: 106346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.691771] LustreError: 106345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.694597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.001247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3579.218970] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3589.615647] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3598.268655] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3605.576415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3605.938563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.869791] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3622.193789] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3626.442573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3636.706532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3636.730532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.743922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3636.753449] Lustre: Skipped 1 previous similar message [ 3636.772837] Lustre: Skipped 1 previous similar message [ 3638.386171] Lustre: server umount lustre-OST0000 complete [ 3645.920412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3645.928239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.943224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3648.991319] Lustre: server umount lustre-MDT0000 complete [ 3652.075955] LustreError: 106342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.114467] LustreError: 106342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3653.253803] LustreError: 106326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786950925 with bad export cookie 6758304886988019704 [ 3653.272989] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3653.274290] LustreError: 106326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3653.734539] Lustre: server umount lustre-MDT0001 complete [ 3666.878491] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3670.287326] Key type lgssc unregistered [ 3670.699498] LNet: 109907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3670.715454] LNetError: 109907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3670.733948] LNet: Removed LNI 192.168.204.105@tcp [ 3671.814145] Key type .llcrypt unregistered [ 3671.818985] Key type ._llcrypt unregistered [ 3700.793979] Key type ._llcrypt registered [ 3700.796292] Key type .llcrypt registered [ 3700.881439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3702.072931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3702.223315] alg: No test for adler32 (adler32-zlib) [ 3703.433297] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3703.751169] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3705.442072] Key type lgssc registered [ 3706.890165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.485789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.500324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.917186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.230148] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3727.337339] LustreError: 111325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.360497] LustreError: 111325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3732.095496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.502897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3737.743380] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3747.286788] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3755.986056] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3765.118086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.756480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.163463] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3784.795284] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3787.315921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3794.533777] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3798.193157] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3804.129304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.152780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.168945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.716226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.720438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.742082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.147525] Lustre: server umount lustre-OST0000 complete [ 3817.953071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.957440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.982432] Lustre: Skipped 1 previous similar message [ 3818.023872] Lustre: Skipped 2 previous similar messages [ 3823.075291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.083975] Lustre: Skipped 1 previous similar message [ 3828.192862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.239216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3830.870800] Lustre: server umount lustre-MDT0000 complete [ 3833.314372] LustreError: 112294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.330066] LustreError: 112294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3835.400806] LustreError: 113171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951108 with bad export cookie 4085062880428624315 [ 3835.415987] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3835.427825] LustreError: 113171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3836.157649] Lustre: server umount lustre-MDT0001 complete [ 3850.542785] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3854.095069] Key type lgssc unregistered [ 3854.496077] LNet: 114972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3854.506199] LNetError: 114972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3854.528446] LNet: Removed LNI 192.168.204.105@tcp [ 3855.681435] Key type .llcrypt unregistered [ 3855.683371] Key type ._llcrypt unregistered [ 3875.371616] Key type ._llcrypt registered [ 3875.373644] Key type .llcrypt registered [ 3875.527805] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3876.565320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3876.747498] alg: No test for adler32 (adler32-zlib) [ 3878.013402] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 3878.500775] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3880.391181] Key type lgssc registered [ 3881.551428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.997771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3896.073944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.953987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.068297] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3908.065567] LustreError: 116195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.101493] LustreError: 116195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3913.186982] LustreError: 116196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.663332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.024863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3919.455176] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3927.538878] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3933.970268] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3941.216157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3941.541368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.579785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3947.275612] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3956.046410] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3970.531362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.539354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.542202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.603254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.607851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.622284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3975.664854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.365172] Lustre: server umount lustre-OST0000 complete [ 3980.771885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.788507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.795071] Lustre: Skipped 2 previous similar messages [ 3985.890068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.911610] Lustre: Skipped 1 previous similar message [ 3986.980483] Lustre: server umount lustre-MDT0000 complete [ 3991.014479] LustreError: 116196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.065405] LustreError: 116196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3992.106116] LustreError: 116177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951264 with bad export cookie 16910198593761882310 [ 3992.136174] LustreError: 116177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3992.137015] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.803368] Lustre: server umount lustre-MDT0001 complete [ 4008.776051] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4011.869947] Key type lgssc unregistered [ 4012.272282] LNet: 119757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4012.280287] LNetError: 119757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4012.309595] LNet: Removed LNI 192.168.204.105@tcp [ 4013.396760] Key type .llcrypt unregistered [ 4013.399419] Key type ._llcrypt unregistered [ 4038.739987] Key type ._llcrypt registered [ 4038.742436] Key type .llcrypt registered [ 4038.851304] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4040.074033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4040.188109] alg: No test for adler32 (adler32-zlib) [ 4041.243989] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 4041.531387] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4043.271213] Key type lgssc registered [ 4044.477994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4052.804198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4052.826755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.294801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.964070] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4063.729513] LustreError: 121174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.763174] LustreError: 121174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4068.302461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.655335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4074.335447] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4083.080731] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4092.077073] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4099.497386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.990722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.278350] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4115.693170] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4123.514196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4135.014683] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4141.038794] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4141.053625] Lustre: Skipped 2 previous similar messages [ 4144.900314] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4144.969209] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4148.292471] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4148.306471] Lustre: Skipped 2 previous similar messages [ 4155.005662] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4155.016967] Lustre: Skipped 2 previous similar messages [ 4161.208621] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4161.220965] Lustre: Skipped 2 previous similar messages [ 4167.291593] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.5@tcp 192.168.0.[2,10]@tcp [ 4172.994993] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.5/24@tcp in log lustre-MDT0000 [ 4173.016769] Lustre: Skipped 2 previous similar messages [ 4176.022413] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.5/24@tcp [ 4176.033241] Lustre: Skipped 1 previous similar message [ 4184.594589] Lustre: server umount lustre-OST0000 complete [ 4190.183070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.193914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4190.203684] Lustre: Skipped 1 previous similar message [ 4195.299513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.302569] Lustre: Skipped 1 previous similar message [ 4198.591121] Lustre: server umount lustre-MDT0000 complete [ 4200.416227] Lustre: 120487:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951457/real 1786951457] req@000000004a2bbcf7 x1873754261099008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786951473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4200.416773] LustreError: 121170:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.453578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.464847] LustreError: 121170:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4201.504245] Lustre: 120487:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786951458/real 1786951458] req@00000000792aeca0 x1873754261100032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786951474 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4203.380276] LustreError: 121153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951476 with bad export cookie 14147780803412887687 [ 4203.385162] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4203.388371] LustreError: 121153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4203.653539] Lustre: server umount lustre-MDT0001 complete [ 4216.891419] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4219.918129] Key type lgssc unregistered [ 4220.441766] LNet: 126101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4220.468702] LNetError: 126101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4220.490292] LNet: Removed LNI 192.168.204.105@tcp [ 4221.599854] Key type .llcrypt unregistered [ 4221.601308] Key type ._llcrypt unregistered [ 4256.655708] Key type ._llcrypt registered [ 4256.668705] Key type .llcrypt registered [ 4256.865872] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4258.613588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4258.709993] alg: No test for adler32 (adler32-zlib) [ 4259.673730] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 4259.974573] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4261.655166] Key type lgssc registered [ 4263.349661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4273.360599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4273.396034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.909650] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4274.941662] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.5/24@tcp [ 4274.963711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.951533] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4284.391856] LustreError: 127661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.418887] LustreError: 127661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4289.513614] LustreError: 127662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.638702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.033624] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4291.066664] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.5/24@tcp [ 4291.126515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4297.187878] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4305.332780] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4313.256237] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4322.703423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4323.261260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.222774] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4339.653731] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4344.828769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4355.047623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.051904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4355.069489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4355.079707] Lustre: Skipped 1 previous similar message [ 4360.160965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.172704] Lustre: Skipped 1 previous similar message [ 4360.653279] Lustre: server umount lustre-OST0000 complete [ 4365.300469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4365.313412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4365.316788] Lustre: Skipped 2 previous similar messages [ 4365.350739] Lustre: Skipped 1 previous similar message [ 4370.404514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.418285] Lustre: Skipped 1 previous similar message [ 4371.182775] Lustre: server umount lustre-MDT0000 complete [ 4375.523103] LustreError: 127662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.566891] LustreError: 127662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4375.609774] LustreError: 127643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951648 with bad export cookie 1308055262567755538 [ 4375.614309] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4375.618607] LustreError: 127643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4376.054338] Lustre: server umount lustre-MDT0001 complete [ 4388.855700] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4391.844366] Key type lgssc unregistered [ 4392.186814] LNet: 131258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4392.199125] LNetError: 131258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4392.232435] LNet: Removed LNI 192.168.204.105@tcp [ 4393.359574] Key type .llcrypt unregistered [ 4393.362233] Key type ._llcrypt unregistered [ 4425.114588] Key type ._llcrypt registered [ 4425.117554] Key type .llcrypt registered [ 4425.268680] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4442.273928] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4443.661763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4443.678138] alg: No test for adler32 (adler32-zlib) [ 4444.868077] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 4445.107621] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4446.799182] Key type lgssc registered [ 4448.058152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4493.233785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4493.267332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.497580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4494.537677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4494.601390] Lustre: lustre-MDT0000: new disk, initializing [ 4494.665467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.683795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4500.513731] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4512.532529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.603719] Lustre: 134660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.640822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4512.643585] Lustre: Skipped 1 previous similar message [ 4512.686058] Lustre: lustre-MDT0001: new disk, initializing [ 4512.804916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4512.864115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4512.879618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4517.179754] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4528.809059] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4536.645902] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4544.879417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.264714] Lustre: lustre-OST0000: new disk, initializing [ 4545.269635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4545.279836] Lustre: 136632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4545.402953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4551.233611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4551.239543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4551.382972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4553.189564] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4563.696723] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4569.295538] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4569.575814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4576.064147] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4576.278971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4582.372086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.387228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.400854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.405096] Lustre: Skipped 1 previous similar message [ 4582.425490] Lustre: Skipped 1 previous similar message [ 4587.492124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4587.509668] Lustre: Skipped 1 previous similar message [ 4592.618612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4592.634240] Lustre: Skipped 1 previous similar message [ 4593.119192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4593.611711] Lustre: server umount lustre-OST0000 complete [ 4599.789885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4599.806596] LustreError: Skipped 1 previous similar message [ 4599.816181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.837397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4602.851448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.731274] Lustre: server umount lustre-MDT0000 complete [ 4611.138632] LustreError: 134655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786951883 with bad export cookie 7924396054680496179 [ 4611.140473] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4611.174745] LustreError: 134655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4611.491286] Lustre: server umount lustre-MDT0001 complete [ 4622.050758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.718540] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.731985] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4622.835066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.613695] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4627.957237] LustreError: 138855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.057019] LustreError: 138854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.181037] LustreError: 138855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.585043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.911140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4644.167564] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4653.320686] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.577227] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4674.007868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.635980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.015415] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4695.959471] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4702.028484] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4702.335782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4708.238293] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4708.728632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4717.534360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4717.819153] Lustre: lustre-OST0001: new disk, initializing [ 4717.827695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4717.854836] Lustre: 142229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4717.969582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4725.934667] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4726.907265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4726.933890] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4727.046285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4736.661362] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4737.002933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4742.327246] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4742.675723] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4749.397658] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff979fd0172800.ost_server_uuid 50 [ 4751.240803] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff979fd0172800.ost_server_uuid in IDLE state after 0 sec [ 4763.108156] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4763.116412] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4763.130861] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4763.130869] Lustre: Skipped 2 previous similar messages [ 4763.161696] Lustre: Skipped 1 previous similar message [ 4767.761243] Lustre: server umount lustre-OST0001 complete [ 4768.255076] LustreError: 140681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4768.281728] LustreError: 140681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4773.347410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.358450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4773.373701] Lustre: Skipped 2 previous similar messages [ 4778.325988] Lustre: server umount lustre-OST0000 complete [ 4783.599728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.614300] Lustre: Skipped 2 previous similar messages [ 4789.133116] Lustre: server umount lustre-MDT0000 complete [ 4793.274066] LustreError: 142228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952065 with bad export cookie 7924396054680497061 [ 4793.282617] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4793.294192] LustreError: 142228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4793.678542] Lustre: server umount lustre-MDT0001 complete [ 4823.626924] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4828.045229] Key type lgssc unregistered [ 4828.396402] LNet: 145238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4828.405095] LNetError: 145238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4828.420496] LNet: Removed LNI 192.168.204.105@tcp [ 4829.705655] Key type .llcrypt unregistered [ 4829.713536] Key type ._llcrypt unregistered [ 4877.420412] Key type ._llcrypt registered [ 4877.423588] Key type .llcrypt registered [ 4877.563495] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4894.439286] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4895.474635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4895.594164] alg: No test for adler32 (adler32-zlib) [ 4896.695820] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 4897.007056] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4898.783229] Key type lgssc registered [ 4900.373727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4939.743748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4939.800846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.188659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4941.239339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4941.357776] Lustre: lustre-MDT0000: new disk, initializing [ 4941.476178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.508244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4947.373202] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4960.239784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4960.403892] Lustre: 149394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4960.462811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4960.466265] Lustre: Skipped 1 previous similar message [ 4960.530420] Lustre: lustre-MDT0001: new disk, initializing [ 4960.647641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4960.707364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4960.716800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4966.199414] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4977.730182] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4984.457948] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4992.770810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4993.245208] Lustre: lustre-OST0000: new disk, initializing [ 4993.250466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4993.262078] Lustre: 151367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4993.432797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.283349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4999.293271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4999.444193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5001.630682] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5013.797420] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5019.816995] Lustre: Failing over lustre-OST0000 [ 5020.008351] Lustre: server umount lustre-OST0000 complete [ 5035.489834] Lustre: 147618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786952292/real 1786952292] req@0000000007b24493 x1873755157597824/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786952308 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5035.494288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.514307] Lustre: 147618:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5036.511983] Lustre: 147619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786952293/real 1786952293] req@000000000b62af3a x1873755157598336/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786952309 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5036.545198] Lustre: 147619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5041.213121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5041.634831] Lustre: 147618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786952297/real 1786952297] req@00000000dec51c3c x1873755157599360/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786952313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5041.679022] Lustre: 147618:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5041.769592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.780753] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5045.863186] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5046.884075] Lustre: 147616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786952303/real 1786952303] req@00000000b89c50c3 x1873755157600512/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786952319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5046.928816] Lustre: 147616:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5047.927768] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5047.928609] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5049.500201] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5054.372645] Lustre: Failing over lustre-MDT0000 [ 5054.641726] Lustre: server umount lustre-MDT0000 complete [ 5056.494095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.518327] Lustre: Skipped 3 previous similar messages [ 5061.601847] LustreError: 149405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.651400] LustreError: 149405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5063.198062] LustreError: 149401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5066.726708] LustreError: 152087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5066.744103] LustreError: 152087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5071.850765] LustreError: 150561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.869763] LustreError: 150561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5072.863938] Lustre: 147618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786952329/real 1786952329] req@0000000022779068 x1873755157611648/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786952345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5072.901643] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.424578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.987625] LustreError: 150561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5078.013631] LustreError: 150561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5083.117018] LustreError: 147615:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002fab4785 x1873755157617792/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5083.431945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.491723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5083.680947] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5088.275173] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5088.746346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5088.754320] Lustre: Skipped 1 previous similar message [ 5088.770889] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5088.833936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5098.979349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5098.984046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5098.997649] Lustre: Skipped 1 previous similar message [ 5099.024566] Lustre: Skipped 1 previous similar message [ 5104.102297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5104.110342] Lustre: Skipped 1 previous similar message [ 5109.220499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.227150] Lustre: Skipped 1 previous similar message [ 5111.263427] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5111.432217] Lustre: server umount lustre-OST0000 complete [ 5119.460508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5119.462908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.492764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.499699] Lustre: Skipped 1 previous similar message [ 5122.007253] Lustre: server umount lustre-MDT0000 complete [ 5124.577223] LustreError: 152087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.590621] LustreError: 152087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5125.995230] LustreError: 150191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952398 with bad export cookie 8048755987282563326 [ 5125.996656] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.018222] LustreError: 150191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5126.509619] Lustre: server umount lustre-MDT0001 complete [ 5141.673219] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5145.356985] Key type lgssc unregistered [ 5145.791689] LNet: 155027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5145.797158] LNetError: 155027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5145.818805] LNet: Removed LNI 192.168.204.105@tcp [ 5147.042986] Key type .llcrypt unregistered [ 5147.046181] Key type ._llcrypt unregistered [ 5191.707670] Key type ._llcrypt registered [ 5191.710646] Key type .llcrypt registered [ 5191.863904] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5193.047280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5193.124065] alg: No test for adler32 (adler32-zlib) [ 5194.322200] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 5194.600882] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5196.287281] Key type lgssc registered [ 5197.633383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5208.343070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5208.375163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.729259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.895387] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5220.321594] LustreError: 156549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.332406] LustreError: 156549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5225.442156] LustreError: 156548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.007762] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.562818] LustreError: 156549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.741753] LustreError: 156577:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5230.759558] LustreError: 156577:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5230.769130] LustreError: 156577:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5230.986945] Lustre: server umount lustre-MDT0000 complete [ 5246.617217] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5257.052442] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5297.769079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.971703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5298.016606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5298.131422] Lustre: lustre-MDT0000: new disk, initializing [ 5298.244790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5298.283597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5303.620939] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5315.196164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.363325] Lustre: 159881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5315.468092] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5315.471788] Lustre: Skipped 1 previous similar message [ 5315.562278] Lustre: lustre-MDT0001: new disk, initializing [ 5315.693341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5315.785663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5315.803128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5321.123581] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5331.253161] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5337.611264] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5345.275161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.567381] Lustre: lustre-OST0000: new disk, initializing [ 5345.574253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5345.582084] Lustre: 161855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5345.647730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.673691] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5353.065463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5353.085506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5353.234927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5362.069311] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5367.318893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5369.126792] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:44:00 (1786952640) [ 5653.473601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5653.480685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5653.497209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.543621] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.546590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.569695] Lustre: Skipped 1 previous similar message [ 5659.785554] Lustre: server umount lustre-MDT0000 complete [ 5661.685574] LustreError: 159892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.705508] LustreError: 159892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5663.488704] LustreError: 161854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786952936 with bad export cookie 17383143740824529249 [ 5663.498883] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.783808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5666.785479] LustreError: 159888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.800773] Lustre: Skipped 1 previous similar message [ 5666.806424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5668.835299] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5670.019452] Lustre: server umount lustre-MDT0001 complete [ 5680.477585] Lustre: server umount lustre-OST0000 complete [ 5690.727227] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5699.520234] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5740.606904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.979349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5741.030517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5741.102405] Lustre: lustre-MDT0000: new disk, initializing [ 5741.277446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.317054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5746.793269] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5758.541420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.725509] Lustre: 166524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5758.790355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5758.794262] Lustre: Skipped 1 previous similar message [ 5758.970290] Lustre: lustre-MDT0001: new disk, initializing [ 5759.126185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.201452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5759.236531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5764.947822] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5778.085402] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5785.713403] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5793.716612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5794.089220] Lustre: lustre-OST0000: new disk, initializing [ 5794.093391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5794.103411] Lustre: 168497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5794.218742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.559855] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5801.578518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5801.590127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5801.700660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5812.200506] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5818.339445] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5818.676452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5823.913786] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5824.270819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5827.556450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5827.558272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.577210] LustreError: Skipped 1 previous similar message [ 5827.585108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5827.596173] Lustre: Skipped 1 previous similar message [ 5827.615337] Lustre: Skipped 1 previous similar message [ 5837.796769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5837.813068] Lustre: Skipped 3 previous similar messages [ 5840.867147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5841.027878] Lustre: server umount lustre-OST0000 complete [ 5845.984488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5845.996585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.755661] Lustre: server umount lustre-MDT0000 complete [ 5853.153357] LustreError: 166536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.167083] LustreError: 166536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5855.864040] LustreError: 166516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953128 with bad export cookie 17383143740824688072 [ 5855.867391] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.873612] LustreError: 166516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5856.328494] Lustre: server umount lustre-MDT0001 complete [ 5867.735448] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:52:18 (1786953138) [ 5886.334753] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5898.967553] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5941.224745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.644294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5941.698993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5941.889227] Lustre: lustre-MDT0000: new disk, initializing [ 5942.077515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5942.117602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5947.738283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5959.742797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.882039] Lustre: 173256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5960.233722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5965.435972] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5976.274902] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5982.443787] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5991.181774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5991.760707] Lustre: lustre-OST0000: new disk, initializing [ 5991.771591] Lustre: Skipped 1 previous similar message [ 5991.779995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5991.786904] Lustre: Skipped 2 previous similar messages [ 5991.796541] Lustre: 175232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5998.169930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5998.181137] Lustre: Skipped 1 previous similar message [ 5998.191442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5998.345775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6000.373949] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6013.150708] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6024.166679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.169279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.180559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.202085] Lustre: Skipped 2 previous similar messages [ 6024.230449] Lustre: Skipped 3 previous similar messages [ 6029.543637] Lustre: server umount lustre-OST0000 complete [ 6040.628587] Lustre: server umount lustre-MDT0000 complete [ 6044.646114] LustreError: 173267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6044.677613] LustreError: 173267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6045.096941] LustreError: 173247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953317 with bad export cookie 17383143740824688961 [ 6045.127211] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6045.799298] Lustre: server umount lustre-MDT0001 complete [ 6056.657847] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:55:27 (1786953327) [ 6076.501548] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 6089.854091] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6133.375291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.683411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6133.702978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6133.795409] Lustre: lustre-MDT0000: new disk, initializing [ 6133.889251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.896134] Lustre: Skipped 2 previous similar messages [ 6133.912816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6140.323165] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6154.042736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.247530] Lustre: 179558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6154.690038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6160.885335] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6174.117588] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6181.945667] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6191.303128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.744529] Lustre: 181537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6199.411965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6199.575467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6199.609924] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6210.964851] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6225.376706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6225.389918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.390935] LustreError: Skipped 2 previous similar messages [ 6225.394541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6225.399484] Lustre: Skipped 3 previous similar messages [ 6225.413879] Lustre: Skipped 7 previous similar messages [ 6226.979293] Lustre: server umount lustre-OST0000 complete [ 6239.064220] Lustre: server umount lustre-MDT0000 complete [ 6240.743251] LustreError: 179566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.771187] LustreError: 179566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6243.324748] LustreError: 181536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953515 with bad export cookie 17383143740824690046 [ 6243.329615] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6243.331984] LustreError: 181536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6243.686606] Lustre: server umount lustre-MDT0001 complete [ 6257.464594] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6261.733906] Key type lgssc unregistered [ 6262.042617] LNet: 183397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6262.057698] LNetError: 183397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6263.083426] LNet: Removed LNI 192.168.204.105@tcp [ 6264.159271] Key type .llcrypt unregistered [ 6264.163465] Key type ._llcrypt unregistered [ 6292.964616] Key type ._llcrypt registered [ 6292.967566] Key type .llcrypt registered [ 6293.188328] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6294.145943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6294.214039] alg: No test for adler32 (adler32-zlib) [ 6295.509891] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6295.950684] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6297.616432] Key type lgssc registered [ 6299.310279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6309.899346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6309.924200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.690413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.489567] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6322.146764] LustreError: 184815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.167880] LustreError: 184815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6327.274918] LustreError: 184814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.270352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6328.925380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6334.677603] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6344.736281] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.326823] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6360.720695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6361.218102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.731520] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6377.545109] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6381.702896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6402.529937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.538538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6402.549716] Lustre: Skipped 1 previous similar message [ 6402.570088] Lustre: Skipped 1 previous similar message [ 6407.539132] Lustre: server umount lustre-OST0000 complete [ 6412.769692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.788435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.792122] Lustre: Skipped 1 previous similar message [ 6412.801537] Lustre: Skipped 1 previous similar message [ 6417.897729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6417.912314] Lustre: Skipped 1 previous similar message [ 6423.019040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6423.029386] Lustre: Skipped 1 previous similar message [ 6426.079820] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6426.263135] Lustre: server umount lustre-MDT0000 complete [ 6428.133617] LustreError: 184810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.159469] LustreError: 184810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6430.721721] LustreError: 184795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953703 with bad export cookie 12143313039396271169 [ 6430.731768] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.741056] LustreError: 184795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6431.125377] Lustre: server umount lustre-MDT0001 complete [ 6443.284813] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6446.708874] Key type lgssc unregistered [ 6447.093891] LNet: 188389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6447.101373] LNetError: 188389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6447.119756] LNet: Removed LNI 192.168.204.105@tcp [ 6448.180527] Key type .llcrypt unregistered [ 6448.183358] Key type ._llcrypt unregistered [ 6477.183581] Key type ._llcrypt registered [ 6477.199267] Key type .llcrypt registered [ 6477.341821] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6479.221616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6479.380360] alg: No test for adler32 (adler32-zlib) [ 6480.719759] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 6481.065068] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6482.784525] Key type lgssc registered [ 6485.156735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6498.108566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6498.127748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.505579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.475424] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6510.048858] LustreError: 189807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.088102] LustreError: 189807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6515.171235] LustreError: 189808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.474485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.899800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6523.293942] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6533.082948] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6540.840948] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6551.790990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6552.383162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.994462] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6569.850784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6571.964863] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6580.201520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6580.202237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.264262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6580.268609] Lustre: Skipped 1 previous similar message [ 6585.353499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.378117] Lustre: Skipped 2 previous similar messages [ 6585.858598] Lustre: server umount lustre-OST0000 complete [ 6595.223465] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff979fc484b800.ost_server_uuid 50 [ 6597.201917] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6614.348375] Lustre: server umount lustre-MDT0000 complete [ 6615.008539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6615.013873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.027823] Lustre: Skipped 1 previous similar message [ 6615.040578] LustreError: 190535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.054891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.614834] LustreError: 189789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786953892 with bad export cookie 17936201482373911048 [ 6619.623396] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6619.638946] LustreError: 189789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6620.340362] Lustre: server umount lustre-MDT0001 complete [ 6631.435474] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:05:01 (1786953901) [ 6640.498941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.059856] LustreError: 193727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6641.101291] LustreError: 193727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6641.195616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.264804] LustreError: 193728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.983212] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6656.481362] LustreError: 193728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6656.508761] LustreError: 193728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6658.160387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.596711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.780360] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6672.158299] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6679.654082] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6689.597979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6689.940598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.063694] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6708.211747] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6710.491399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6717.533710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.801611] Lustre: lustre-OST0001: new disk, initializing [ 6717.804900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6717.813541] Lustre: 196573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6717.981075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6723.256423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6723.269381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6723.454789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6728.182538] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6739.893561] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6744.048856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.058165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.060943] Lustre: Skipped 1 previous similar message [ 6749.158586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6749.187130] Lustre: Skipped 3 previous similar messages [ 6749.419791] Lustre: server umount lustre-OST0000 complete [ 6750.759124] LustreError: 195553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6758.099483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6758.396727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6765.411485] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6765.716624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6784.993144] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6784.995773] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.999850] LustreError: 197381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.999862] LustreError: 197381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6785.005589] LustreError: Skipped 1 previous similar message [ 6785.011441] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6785.027805] Lustre: Skipped 2 previous similar messages [ 6785.096604] Lustre: Skipped 1 previous similar message [ 6788.089140] Lustre: server umount lustre-OST0001 complete [ 6793.184678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.200355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6793.208504] Lustre: Skipped 1 previous similar message [ 6801.428919] Lustre: server umount lustre-MDT0000 complete [ 6806.653493] LustreError: 193707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954079 with bad export cookie 17936201482373912119 [ 6806.663068] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6806.686333] LustreError: 193707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6807.779486] Lustre: server umount lustre-MDT0001 complete [ 6842.408438] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:08:32 (1786954112) [ 6851.567208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.064064] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6852.124535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6852.339656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6857.945661] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6870.709521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.815894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6870.891224] Lustre: 200014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6877.416137] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6887.905435] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6895.384366] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6902.920409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.106558] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6908.592558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6910.942223] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6921.117041] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6930.712745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.922701] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6931.208519] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6931.212117] Lustre: Skipped 2 previous similar messages [ 6939.704810] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6940.987658] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6949.302576] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.467798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6954.472266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.492721] Lustre: Skipped 1 previous similar message [ 6954.498551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6954.510690] Lustre: Skipped 2 previous similar messages [ 6959.254981] Lustre: server umount lustre-OST0000 complete [ 6960.674788] LustreError: 203293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.700561] LustreError: 203293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6979.306723] Lustre: server umount lustre-OST0001 complete [ 6984.162689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6984.177352] LustreError: Skipped 1 previous similar message [ 6987.233907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.241583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6987.266543] Lustre: Skipped 2 previous similar messages [ 6987.277896] Lustre: Skipped 6 previous similar messages [ 6993.446987] Lustre: server umount lustre-MDT0000 complete [ 6997.254891] LustreError: 200007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954269 with bad export cookie 17936201482373913526 [ 6997.261944] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.268986] LustreError: 200007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7006.980245] Lustre: server umount lustre-MDT0001 complete [ 7036.585502] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:11:47 (1786954307) [ 7057.951978] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 7070.575923] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7119.422043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7119.756950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7119.795759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7119.960364] Lustre: lustre-MDT0000: new disk, initializing [ 7120.115096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.142077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7125.993311] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7138.285057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7138.436840] Lustre: 208124:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7138.528658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7138.540278] Lustre: Skipped 1 previous similar message [ 7138.668420] Lustre: lustre-MDT0001: new disk, initializing [ 7138.859318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7138.876040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7145.108434] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7157.684727] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7164.911991] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7173.657286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.055392] Lustre: lustre-OST0000: new disk, initializing [ 7174.059765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7174.066968] Lustre: 210094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7181.582591] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7182.943647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7182.971428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7183.105671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7192.930188] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.222885] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7199.608863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7204.690753] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7204.936916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7208.931875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7208.932594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7208.959152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7208.964279] Lustre: Skipped 1 previous similar message [ 7208.981687] Lustre: Skipped 3 previous similar messages [ 7214.350690] Lustre: server umount lustre-OST0000 complete [ 7225.346737] Lustre: server umount lustre-MDT0000 complete [ 7229.408282] LustreError: 208799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.420818] LustreError: 208799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7229.732367] LustreError: 208117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954502 with bad export cookie 17936201482373915311 [ 7229.742784] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7229.753503] LustreError: 208117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7230.002119] Lustre: server umount lustre-MDT0001 complete [ 7241.091741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.047985] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7257.538291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.766707] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7272.660373] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7279.669362] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7287.960368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.389550] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7306.237200] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7311.519333] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7312.817273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7318.027964] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7318.214235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7326.893763] Lustre: server umount lustre-OST0000 complete [ 7334.925275] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7340.813798] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7371.403058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.830925] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7386.671451] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7388.341086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7388.348222] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7388.353416] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7388.363981] Lustre: Skipped 1 previous similar message [ 7388.451569] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7393.762385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7393.765284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.768027] LustreError: Skipped 1 previous similar message [ 7393.775517] Lustre: Skipped 4 previous similar messages [ 7393.777100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7393.784121] Lustre: Skipped 9 previous similar messages [ 7399.776753] Lustre: server umount lustre-OST0000 complete [ 7419.362700] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7421.523689] Lustre: server umount lustre-MDT0000 complete [ 7425.489351] LustreError: 212295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954698 with bad export cookie 17936201482373916193 [ 7425.499097] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7425.501499] LustreError: 212295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7425.863514] Lustre: server umount lustre-MDT0001 complete [ 7437.611682] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:18:28 (1786954708) [ 7445.612787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.188104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.197801] Lustre: Skipped 6 previous similar messages [ 7451.475642] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7460.909635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.792602] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7474.760430] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7481.776680] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7489.342287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.974966] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7505.808519] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7512.178775] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7512.537781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7517.773913] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7518.096463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7525.856153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7526.057273] Lustre: lustre-OST0001: new disk, initializing [ 7526.061512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7526.067938] Lustre: 221661:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7534.263973] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7535.735961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7535.752795] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7535.847987] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7545.174935] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7550.346673] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7550.586508] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7555.616444] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7555.891741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7561.708852] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7561.717266] LustreError: Skipped 1 previous similar message [ 7564.554947] Lustre: server umount lustre-OST0001 complete [ 7566.825316] LustreError: 222091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.834957] LustreError: 222091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7571.981878] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7572.433386] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7579.135948] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7579.380563] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7610.184609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.470420] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7615.498461] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7615.501032] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7615.529788] Lustre: Skipped 1 previous similar message [ 7617.914173] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7626.604133] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7628.431371] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7630.830258] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7636.578825] Lustre: server umount lustre-OST0001 complete [ 7654.881272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.891084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.897293] Lustre: Skipped 8 previous similar messages [ 7654.920121] Lustre: Skipped 16 previous similar messages [ 7669.215242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7669.373496] Lustre: server umount lustre-OST0000 complete [ 7681.518651] Lustre: server umount lustre-MDT0000 complete [ 7685.983186] LustreError: 218273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786954958 with bad export cookie 17936201482373917159 [ 7685.990878] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7685.993453] LustreError: 218273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7692.458458] Lustre: server umount lustre-MDT0001 complete [ 7720.916634] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:23:11 (1786954991) [ 7728.035121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.256945] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7728.277907] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7732.828874] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7742.565265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.680631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7742.716742] Lustre: 226911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7748.004833] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7756.968746] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7763.464951] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7770.852517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.095242] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7778.289285] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7787.015445] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7796.739833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7796.805401] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7803.974528] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7812.593347] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7814.964827] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7821.128861] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7821.449050] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7826.676706] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7826.939318] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7833.636689] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff979fc8f97000.ost_server_uuid 50 [ 7836.121174] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff979fc8f97000.ost_server_uuid in IDLE state after 0 sec [ 7837.712270] Lustre: Permanently deactivating lustre-OST0001 [ 7837.754112] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7842.990504] Lustre: Permanently reactivating lustre-OST0001 [ 7843.016523] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7843.019162] Lustre: Skipped 2 previous similar messages [ 7848.939284] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7848.944695] LustreError: Skipped 4 previous similar messages [ 7853.760100] Lustre: server umount lustre-OST0001 complete [ 7864.938991] Lustre: server umount lustre-OST0000 complete [ 7875.878048] Lustre: server umount lustre-MDT0000 complete [ 7881.023939] LustreError: 226905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955153 with bad export cookie 17936201482373918447 [ 7881.033224] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7881.040390] LustreError: 226905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7881.541092] Lustre: server umount lustre-MDT0001 complete [ 7908.659841] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:26:19 (1786955179) [ 7917.761140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.215956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7918.254531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7918.262675] Lustre: Skipped 2 previous similar messages [ 7924.070501] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7935.703773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.839209] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7935.870939] Lustre: 233612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7942.782898] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7952.719770] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7959.459387] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7967.870271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7968.107050] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7968.132626] Lustre: Permanently deactivating lustre-OST0000 [ 7968.145356] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7968.327155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.334405] Lustre: Skipped 10 previous similar messages [ 7975.392870] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7977.522725] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7977.537209] LustreError: 235955:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 7983.905641] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7991.367520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7991.455714] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7999.774934] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8000.192985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8008.556235] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8010.819695] Lustre: Permanently reactivating lustre-OST0000 [ 8010.833454] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8010.838873] Lustre: Skipped 2 previous similar messages [ 8017.483476] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8017.485957] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 00000000ae91911f, cur 1786955290 deadline 1786955280 last 1786955250 [ 8017.489243] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8017.490275] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8017.517991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8017.579857] LustreError: 235959:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -107 [ 8018.598365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8035.636932] Lustre: server umount lustre-OST0001 complete [ 8054.239285] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8054.547422] Lustre: server umount lustre-OST0000 complete [ 8066.198996] Lustre: server umount lustre-MDT0000 complete [ 8071.516339] LustreError: 237209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955344 with bad export cookie 17936201482373920113 [ 8071.535603] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8071.536755] LustreError: 237209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8072.060818] Lustre: server umount lustre-MDT0001 complete [ 8085.647049] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 8089.294720] Key type lgssc unregistered [ 8089.637932] LNet: 238500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8089.649586] LNetError: 238500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8089.668202] LNet: Removed LNI 192.168.204.105@tcp [ 8090.776173] Key type .llcrypt unregistered [ 8090.778610] Key type ._llcrypt unregistered [ 8117.209667] Key type ._llcrypt registered [ 8117.216268] Key type .llcrypt registered [ 8117.347962] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 8118.779420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8118.941462] alg: No test for adler32 (adler32-zlib) [ 8120.007838] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 8120.382861] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 8122.144404] Key type lgssc registered [ 8123.334639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8135.758987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8135.800915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.484619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8143.970662] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8147.949113] LustreError: 239972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8147.964872] LustreError: 239972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8153.071945] LustreError: 239973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.467037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.640247] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8155.671264] Lustre: 239959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8155.682426] Lustre: Permanently deactivating lustre-MDT0001 [ 8155.703236] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8156.116502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8162.757618] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8164.401748] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8173.467989] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8181.534567] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8189.797692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8190.366673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8197.378196] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8201.661820] LustreError: 241814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8201.696833] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 8206.826096] LustreError: 241816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.310856] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8207.870471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8212.965983] LustreError: 242483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8212.999514] LustreError: 242483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8216.493331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.895381] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.207971] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8222.217645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8224.961909] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8233.869162] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8238.142743] Lustre: Permanently reactivating lustre-MDT0001 [ 8238.164456] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8238.169979] Lustre: Skipped 2 previous similar messages [ 8241.960476] Lustre: lustre-MDT0001: Client 77cd8c4c-a962-40b3-ac79-6e053990d45c (at 192.168.204.5@tcp) reconnecting [ 8243.846709] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.866043] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8243.878151] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8243.902459] LustreError: 243544:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 8243.903732] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8243.911541] LustreError: 243544:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 8243.911678] LustreError: 243544:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.105@tcp: cfg command failed: rc = -108 [ 8243.911703] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 8243.911703] [ 8243.912379] LustreError: 239964:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 8250.019261] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.224457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8258.931039] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8259.171764] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8263.068403] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8263.237322] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8266.437884] Lustre: Permanently deactivating lustre-MDT0001 [ 8266.465332] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8266.475692] Lustre: Skipped 2 previous similar messages [ 8272.051810] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8284.646113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8284.657529] Lustre: Skipped 1 previous similar message [ 8284.661225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8284.669966] Lustre: Skipped 2 previous similar messages [ 8289.248639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8289.255146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.272366] Lustre: Skipped 1 previous similar message [ 8289.277430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.775147] Lustre: server umount lustre-MDT0000 complete [ 8294.886022] LustreError: 239972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.922450] LustreError: 239972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8295.348348] LustreError: 239954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955567 with bad export cookie 16644268901730883261 [ 8295.351798] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8295.367263] LustreError: 239954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8295.774704] Lustre: server umount lustre-MDT0001 complete [ 8302.543660] Lustre: server umount lustre-OST0000 complete [ 8315.748260] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:33:06 (1786955586) [ 8327.658339] Lustre: server umount lustre-OST0001 complete [ 8335.244464] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 8346.994588] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 8387.944397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8388.187925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8388.195597] Lustre: Skipped 2 previous similar messages [ 8388.215842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8388.290095] Lustre: lustre-MDT0000: new disk, initializing [ 8388.357825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.373528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8393.319718] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8403.827796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.013358] Lustre: 248257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8404.115801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8404.118223] Lustre: Skipped 1 previous similar message [ 8404.177244] Lustre: lustre-MDT0001: new disk, initializing [ 8404.268224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8404.371907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8404.389689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8410.016121] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8421.938855] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8429.699912] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8436.665602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.962467] Lustre: lustre-OST0000: new disk, initializing [ 8436.973899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8436.987974] Lustre: 250229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8437.041555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8442.828750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8446.544171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8446.559996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8446.715902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8452.728171] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8455.237404] LustreError: 248264:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8465.264086] LustreError: 248264:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8465.298686] LustreError: 249871:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8465.827700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8466.038537] Lustre: lustre-OST0001: new disk, initializing [ 8466.042110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8466.052564] Lustre: 251344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8474.252491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8474.277176] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8474.407830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8474.828969] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8475.359401] LustreError: 249871:0:(mdt_handler.c:3276:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8486.051939] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8487.883596] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8495.077254] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8495.082403] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8495.083273] LustreError: Skipped 1 previous similar message [ 8495.084544] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8495.096524] Lustre: Skipped 1 previous similar message [ 8495.120223] Lustre: Skipped 4 previous similar messages [ 8496.569449] Lustre: server umount lustre-OST0001 complete [ 8500.199091] LustreError: 250211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.221285] LustreError: 250211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8515.566312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.571738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8515.580216] Lustre: Skipped 1 previous similar message [ 8520.692677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8520.698502] Lustre: Skipped 2 previous similar messages [ 8529.377450] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8529.512175] Lustre: server umount lustre-OST0000 complete [ 8536.039769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.043589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8536.062932] Lustre: Skipped 1 previous similar message [ 8536.074228] Lustre: Skipped 2 previous similar messages [ 8541.060055] Lustre: server umount lustre-MDT0000 complete [ 8541.158097] LustreError: 251555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.173398] LustreError: 251555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8545.607053] LustreError: 248250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786955818 with bad export cookie 16644268901730886299 [ 8545.612879] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8545.621039] LustreError: 248250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8552.303311] Lustre: server umount lustre-MDT0001 complete [ 8566.513265] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 8569.577810] Key type lgssc unregistered [ 8569.909673] LNet: 253383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8569.917325] LNetError: 253383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8569.934644] LNet: Removed LNI 192.168.204.105@tcp [ 8570.841736] Key type .llcrypt unregistered [ 8570.843828] Key type ._llcrypt unregistered [ 8616.397717] Key type ._llcrypt registered [ 8616.399758] Key type .llcrypt registered [ 8616.500483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 8617.840158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8617.904418] alg: No test for adler32 (adler32-zlib) [ 8618.908811] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 8619.061448] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 8620.672083] Key type lgssc registered [ 8622.335591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8632.379589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8632.394540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8633.761812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8633.798237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8634.029047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8640.173677] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8651.358487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.468588] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8651.532880] Lustre: 255597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8652.065523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8658.748597] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8668.238033] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8675.174578] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8683.274269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8683.564386] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8683.930560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8690.907803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8692.028921] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8702.618456] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8762.485264] Lustre: server umount lustre-OST0000 complete [ 8768.664973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8778.703994] Lustre: 254921:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786956035/real 1786956035] req@00000000d6f9ac2d x1873759060411264/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786956051 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8778.739990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.227763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.342755] Lustre: 254919:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786956040/real 1786956040] req@000000006fa1924a x1873759060412928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786956056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8784.377727] Lustre: 254919:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8784.786319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8788.902184] Lustre: 254920:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786956045/real 1786956045] req@00000000c9b3c27b x1873759060414208/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786956061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8788.929092] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8788.945447] Lustre: 254920:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8788.984531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8789.003117] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8791.490807] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8800.838421] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8845.290656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.319536] Lustre: Skipped 1 previous similar message [ 8845.330446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8845.331058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8847.945470] Lustre: server umount lustre-OST0000 complete [ 8855.521133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.524782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8855.553315] Lustre: Skipped 2 previous similar messages [ 8855.583481] Lustre: Skipped 2 previous similar messages [ 8859.538039] Lustre: server umount lustre-MDT0000 complete [ 8864.489673] LustreError: 255589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956137 with bad export cookie 9163865467787005836 [ 8864.492525] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8864.505969] LustreError: 255589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8865.089780] Lustre: server umount lustre-MDT0001 complete [ 8880.515153] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 8883.753425] Key type lgssc unregistered [ 8884.148905] LNet: 261028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8884.154439] LNetError: 261028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8885.222684] LNet: Removed LNI 192.168.204.105@tcp [ 8886.166739] Key type .llcrypt unregistered [ 8886.168764] Key type ._llcrypt unregistered [ 8920.381849] Key type ._llcrypt registered [ 8920.389920] Key type .llcrypt registered [ 8920.501611] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 8922.192713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8922.235468] alg: No test for adler32 (adler32-zlib) [ 8923.421302] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 8923.616088] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 8925.240214] Key type lgssc registered [ 8926.290935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8935.708816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8935.738553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.249015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8943.210598] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8946.660793] LustreError: 262452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.678703] LustreError: 262452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8951.783904] LustreError: 262451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8953.789976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.306635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8960.232876] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8970.473315] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8978.150925] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8986.665262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8987.141696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8994.889741] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9003.197638] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9006.642995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 9032.161576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9032.166656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9032.177904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9032.201619] Lustre: Skipped 1 previous similar message [ 9037.299684] Lustre: server umount lustre-OST0000 complete [ 9042.404821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.418409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9042.422034] Lustre: Skipped 1 previous similar message [ 9042.439603] Lustre: Skipped 2 previous similar messages [ 9047.522036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9047.533047] Lustre: Skipped 1 previous similar message [ 9047.913194] Lustre: server umount lustre-MDT0000 complete [ 9051.624672] LustreError: 262448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9051.651539] LustreError: 262448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9051.800503] LustreError: 263186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956324 with bad export cookie 8428557509541669698 [ 9051.803201] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9051.810774] LustreError: 263186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9052.101350] Lustre: server umount lustre-MDT0001 complete [ 9063.754173] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9067.342865] Key type lgssc unregistered [ 9067.733559] LNet: 266759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9067.757613] LNetError: 266759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9068.847179] LNet: Removed LNI 192.168.204.105@tcp [ 9069.767835] Key type .llcrypt unregistered [ 9069.771661] Key type ._llcrypt unregistered [ 9088.489171] Key type ._llcrypt registered [ 9088.493794] Key type .llcrypt registered [ 9088.605232] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9089.627454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9089.661265] alg: No test for adler32 (adler32-zlib) [ 9090.660227] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 9090.885324] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9092.560238] Key type lgssc registered [ 9093.991905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9101.456815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9101.492716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.832668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9107.161723] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9113.069701] LustreError: 267982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9113.084893] LustreError: 267982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9116.386510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9116.786414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9121.324103] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9128.683846] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9134.347985] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9141.195653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9141.709715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.748970] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9147.904196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9155.782979] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9172.454163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9172.463257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.479165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.476810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.480950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9173.498795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.808775] Lustre: server umount lustre-OST0000 complete [ 9183.201359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9183.212132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.236282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.244803] Lustre: Skipped 1 previous similar message [ 9185.248282] Lustre: server umount lustre-MDT0000 complete [ 9188.845306] LustreError: 267982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9188.872937] LustreError: 267982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9189.285881] LustreError: 269825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956461 with bad export cookie 5461584824677945869 [ 9189.288759] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9189.309275] LustreError: 269825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9189.764749] Lustre: server umount lustre-MDT0001 complete [ 9200.190284] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9202.840958] Key type lgssc unregistered [ 9203.077780] LNet: 271663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9203.084344] LNetError: 271663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9203.103524] LNet: Removed LNI 192.168.204.105@tcp [ 9203.921374] Key type .llcrypt unregistered [ 9203.926461] Key type ._llcrypt unregistered [ 9227.747737] Key type ._llcrypt registered [ 9227.750706] Key type .llcrypt registered [ 9227.831995] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9228.791882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.934770] alg: No test for adler32 (adler32-zlib) [ 9230.004556] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 9230.250978] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9231.983382] Key type lgssc registered [ 9233.245410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9242.236856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9242.261668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.769575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9248.250489] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9254.386852] LustreError: 273085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.403037] LustreError: 273085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9256.974738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9257.311568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9262.682912] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9271.007521] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9277.585638] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9285.685673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.105994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9290.251737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9293.470735] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9301.114728] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9326.053539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9326.060450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9326.083392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9327.075531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9327.077823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9327.102975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9327.126961] Lustre: Skipped 1 previous similar message [ 9329.865602] Lustre: server umount lustre-OST0000 complete [ 9334.240382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9334.253139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9334.269522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.313833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9337.316427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.339486] Lustre: Skipped 1 previous similar message [ 9340.587501] Lustre: server umount lustre-MDT0000 complete [ 9342.432570] LustreError: 273079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.451715] LustreError: 273079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9345.098915] LustreError: 273065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956617 with bad export cookie 1765863832117633473 [ 9345.100386] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9345.107860] LustreError: 273065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9345.488718] Lustre: server umount lustre-MDT0001 complete [ 9357.098463] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9360.501616] Key type lgssc unregistered [ 9360.891035] LNet: 277394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9360.901273] LNetError: 277394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9360.914755] LNet: Removed LNI 192.168.204.105@tcp [ 9362.002313] Key type .llcrypt unregistered [ 9362.005633] Key type ._llcrypt unregistered [ 9379.446976] Key type ._llcrypt registered [ 9379.449902] Key type .llcrypt registered [ 9379.561495] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9381.077617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9381.140694] alg: No test for adler32 (adler32-zlib) [ 9382.429713] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 9382.759559] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9384.481749] Key type lgssc registered [ 9385.887362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9394.589618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9394.621637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.003911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9400.952038] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9405.410486] LustreError: 278632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.440361] LustreError: 278632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9410.538590] LustreError: 278634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.659024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.086920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9415.429717] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9423.424127] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9428.767903] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9436.111136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9436.531770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.639813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9443.243681] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9450.875315] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9466.346579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9466.353082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.359397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9466.373526] Lustre: Skipped 1 previous similar message [ 9466.396945] Lustre: Skipped 1 previous similar message [ 9471.456723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9471.467681] Lustre: Skipped 1 previous similar message [ 9471.726628] Lustre: server umount lustre-OST0000 complete [ 9476.578157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.580191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9476.604884] Lustre: Skipped 1 previous similar message [ 9481.696597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9481.710742] Lustre: Skipped 2 previous similar messages [ 9481.933867] Lustre: server umount lustre-MDT0000 complete [ 9485.848837] LustreError: 278599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956758 with bad export cookie 14314561979578536145 [ 9485.857359] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9485.869819] LustreError: 278599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9486.248071] Lustre: server umount lustre-MDT0001 complete [ 9498.140281] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9500.594053] Key type lgssc unregistered [ 9500.896595] LNet: 282334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9500.903696] LNetError: 282334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9500.921866] LNet: Removed LNI 192.168.204.105@tcp [ 9501.862460] Key type .llcrypt unregistered [ 9501.869684] Key type ._llcrypt unregistered [ 9514.049631] Key type ._llcrypt registered [ 9514.051754] Key type .llcrypt registered [ 9514.140706] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9530.422159] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 9546.070748] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9547.715849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9547.749352] alg: No test for adler32 (adler32-zlib) [ 9548.846683] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 9549.076178] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9550.784657] Key type lgssc registered [ 9551.932366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9588.581327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9588.626697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.868074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9589.907531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9590.025645] Lustre: lustre-MDT0000: new disk, initializing [ 9590.202193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.224803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9595.054808] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9606.610500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9606.755794] Lustre: 285886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9606.823943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9606.828091] Lustre: Skipped 1 previous similar message [ 9606.930428] Lustre: lustre-MDT0001: new disk, initializing [ 9607.074120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9607.128461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9607.142145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9612.499475] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9622.998138] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9629.357511] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9637.002400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.494652] Lustre: lustre-OST0000: new disk, initializing [ 9637.499972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9637.510134] Lustre: 287855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9637.664251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9643.613532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9643.631681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9643.759174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9645.282640] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9655.541079] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9660.381680] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9660.629849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9664.572226] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9664.758730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9669.603167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.605062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9669.606812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9669.627643] Lustre: Skipped 1 previous similar message [ 9669.664240] LustreError: Skipped 1 previous similar message [ 9672.866819] Lustre: server umount lustre-OST0000 complete [ 9678.816710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9678.830153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.855988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9678.862256] Lustre: Skipped 1 previous similar message [ 9679.854832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.875766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9679.887539] Lustre: Skipped 1 previous similar message [ 9684.960274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9684.974674] Lustre: Skipped 1 previous similar message [ 9690.082818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9690.100882] Lustre: Skipped 1 previous similar message [ 9691.104787] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9691.284039] Lustre: server umount lustre-MDT0000 complete [ 9695.315853] LustreError: 285880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786956967 with bad export cookie 14274726281894121082 [ 9695.326563] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9695.707981] Lustre: server umount lustre-MDT0001 complete [ 9705.504894] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:56:16 (1786956976) [ 9712.123134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.735666] LustreError: 290270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9712.763865] LustreError: 290270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9712.828837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.241263] LustreError: 290271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.414761] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9723.368030] LustreError: 290270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.584081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.893636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9730.828352] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9737.793475] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9743.603264] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9750.557901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9750.893331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9756.832402] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9764.007458] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9774.563642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9774.576433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9774.600262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9774.614715] Lustre: Skipped 1 previous similar message [ 9778.868435] Lustre: server umount lustre-OST0000 complete [ 9783.272970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9783.288658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9783.304643] Lustre: Skipped 1 previous similar message [ 9795.045403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9795.049511] Lustre: Skipped 6 previous similar messages [ 9797.089303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9797.234878] Lustre: server umount lustre-MDT0000 complete [ 9800.164197] LustreError: 290266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9800.181883] LustreError: 290266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9800.848923] LustreError: 292766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957073 with bad export cookie 14274726281894121964 [ 9800.852813] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.857784] LustreError: 292766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9801.136682] Lustre: server umount lustre-MDT0001 complete [ 9812.179705] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9815.243280] Key type lgssc unregistered [ 9815.509895] LNet: 293810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9815.516737] LNetError: 293810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9815.541352] LNet: Removed LNI 192.168.204.105@tcp [ 9816.331376] Key type .llcrypt unregistered [ 9816.334813] Key type ._llcrypt unregistered [ 9843.573099] Key type ._llcrypt registered [ 9843.574774] Key type .llcrypt registered [ 9843.724684] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 9859.885398] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9860.869823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9861.027600] alg: No test for adler32 (adler32-zlib) [ 9862.002212] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [ 9862.365730] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9864.072040] Key type lgssc registered [ 9865.505939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9916.389894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9916.426399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9917.691953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9917.728420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9917.800519] Lustre: lustre-MDT0000: new disk, initializing [ 9917.858495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.873838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9922.499892] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9932.899760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.033696] Lustre: 297657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9933.084317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9933.088281] Lustre: Skipped 1 previous similar message [ 9933.150446] Lustre: lustre-MDT0001: new disk, initializing [ 9933.266256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9933.321310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9933.340137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9937.772444] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9947.736988] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9954.580137] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9961.752922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9962.071163] Lustre: lustre-OST2710: new disk, initializing [ 9962.074626] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9962.089267] Lustre: 299623:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9962.184909] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9967.170801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9967.184577] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9967.294440] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9968.415275] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9977.932773] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9984.939099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9985.068824] Lustre: lustre-OST03e8: new disk, initializing [ 9985.078117] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9985.083523] Lustre: 300685:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9985.173388] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9991.719433] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9991.818745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9991.829477] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9991.931069] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9999.370989] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10005.754816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10005.947675] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10010.688700] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10010.901551] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10015.886295] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10016.102645] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10020.560544] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10020.726655] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10025.447822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10025.455502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.487060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.004331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.004408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.010882] Lustre: Skipped 1 previous similar message [10028.024298] Lustre: Skipped 2 previous similar messages [10031.268759] Lustre: server umount lustre-MDT0000 complete [10035.277454] LustreError: 297647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957307 with bad export cookie 4881960643237209162 [10035.282990] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10035.286506] LustreError: 297647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10035.696179] Lustre: server umount lustre-MDT0001 complete [10050.180499] Lustre: server umount lustre-OST2710 complete [10054.093786] Lustre: 295453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786957310/real 1786957310] req@00000000675faa0e x1873760363809664/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786957326 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [10064.577483] Lustre: server umount lustre-OST03e8 complete [10083.987299] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [10096.879965] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10136.795402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10137.043276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10137.075792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10137.137102] Lustre: lustre-MDT0000: new disk, initializing [10137.248090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10137.263698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10141.698252] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10153.691043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.828422] Lustre: 305951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10153.877988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10153.883063] Lustre: Skipped 1 previous similar message [10154.020554] Lustre: lustre-MDT0001: new disk, initializing [10154.188375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10154.237411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10154.249795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10159.170267] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10169.941389] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10177.139551] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10184.053048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10184.288306] Lustre: lustre-OST0000: new disk, initializing [10184.290221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10184.301570] Lustre: 307922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10184.407541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10190.820609] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10192.451502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10192.459826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10192.545988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10200.479094] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10205.133556] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10205.387286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10209.510599] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10209.688104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10213.345544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.360908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10213.394432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10213.419815] Lustre: Skipped 3 previous similar messages [10217.717597] Lustre: server umount lustre-OST0000 complete [10223.585348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10223.604460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10223.619428] Lustre: Skipped 2 previous similar messages [10223.653419] Lustre: Skipped 1 previous similar message [10228.707783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10228.719757] Lustre: Skipped 1 previous similar message [10237.408255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10237.701962] Lustre: server umount lustre-MDT0000 complete [10238.944457] LustreError: 305955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10238.954771] LustreError: 305955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10242.119661] LustreError: 305941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957514 with bad export cookie 4881960643237210842 [10242.124225] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.145035] LustreError: 305941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10242.409595] Lustre: server umount lustre-MDT0001 complete [10254.176990] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:05:25 (1786957525) [10256.399481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10259.266532] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:05:29 (1786957529) [10263.471231] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10293.113960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.475829] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10293.556835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10293.766084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10298.867548] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10307.816495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.943839] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10307.988313] Lustre: 311778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10312.881155] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10334.453429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10334.608300] Lustre: Denying initial registration attempt from nid 192.168.204.105@tcp, specified as failover [10334.617728] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10334.626379] LustreError: 313231:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10334.636274] LustreError: 313231:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10334.645879] LustreError: 313231:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10334.716622] Lustre: server umount lustre-OST0000 complete [10334.719068] LustreError: 313231:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10344.417888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.419360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10344.420288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10344.420293] Lustre: Skipped 2 previous similar messages [10344.425855] Lustre: Skipped 1 previous similar message [10344.446778] LustreError: Skipped 1 previous similar message [10347.230623] Lustre: server umount lustre-MDT0000 complete [10349.546716] LustreError: 313232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10349.572026] LustreError: 313232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10350.987299] LustreError: 311770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957623 with bad export cookie 4881960643237211724 [10350.995051] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10351.000135] LustreError: 311770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10351.338043] Lustre: server umount lustre-MDT0001 complete [10362.752366] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10365.416206] Key type lgssc unregistered [10365.715544] LNet: 314356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10365.724901] LNetError: 314356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10365.742437] LNet: Removed LNI 192.168.204.105@tcp [10366.593285] Key type .llcrypt unregistered [10366.595809] Key type ._llcrypt unregistered [10378.164149] Key type ._llcrypt registered [10378.168481] Key type .llcrypt registered [10378.286680] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10379.337313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10379.404306] alg: No test for adler32 (adler32-zlib) [10380.463737] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [10380.739612] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10382.415166] Key type lgssc registered [10383.495541] Lustre: Echo OBD driver; http://www.lustre.org/ [10426.446868] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10435.113403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10435.141526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10436.536832] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10436.570236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10436.755853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10441.537575] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10451.421390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.490994] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10451.562991] Lustre: 317122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10451.901067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10457.506227] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10466.299735] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10472.254611] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10492.528606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.751209] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10492.919653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10498.845484] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10507.481856] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10513.377680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10513.385261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.403347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10513.888495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10513.898722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.913260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10515.827772] Lustre: server umount lustre-OST0000 complete [10523.631369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10523.640550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.659647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10525.893761] Lustre: server umount lustre-MDT0000 complete [10529.296380] LustreError: 317873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957801 with bad export cookie 6096250888687236278 [10529.299227] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10529.308019] LustreError: 317873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10529.571483] Lustre: server umount lustre-MDT0001 complete [10541.564550] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10544.543797] Key type lgssc unregistered [10544.940810] LNet: 320701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10544.950303] LNetError: 320701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10544.996822] LNet: Removed LNI 192.168.204.105@tcp [10546.237601] Key type .llcrypt unregistered [10546.239848] Key type ._llcrypt unregistered [10575.209809] Key type ._llcrypt registered [10575.214274] Key type .llcrypt registered [10575.332857] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10577.173159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10577.208118] alg: No test for adler32 (adler32-zlib) [10578.378095] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [10578.661524] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10580.407351] Key type lgssc registered [10581.989068] Lustre: Echo OBD driver; http://www.lustre.org/ [10591.877476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10591.899182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.518433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10598.396170] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10604.021804] LustreError: 322120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10604.045780] LustreError: 322120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10609.127820] LustreError: 322121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10609.954439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.713502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10616.450399] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10626.770709] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10634.197501] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10644.262586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10644.616086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10650.771653] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10659.142461] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10674.663810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10674.667855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10674.685876] Lustre: Skipped 2 previous similar messages [10674.694188] Lustre: Skipped 1 previous similar message [10676.296511] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [10676.309832] Lustre: Skipped 1 previous similar message [10679.783545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10679.795968] Lustre: Skipped 2 previous similar messages [10680.382606] Lustre: server umount lustre-MDT0000 complete [10681.389252] LustreError: 323361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.911753] LustreError: 322115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.929603] LustreError: 322115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10685.008690] LustreError: 322100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786957957 with bad export cookie 13528074887676587965 [10685.014707] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10685.023576] LustreError: 322100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10686.542276] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [10690.020083] LustreError: 322116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10690.022599] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10690.044736] LustreError: 322116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10691.468570] Lustre: server umount lustre-MDT0001 complete [10698.545460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10709.400025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10710.816548] LustreError: 323953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.969228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10711.045591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10716.085746] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10720.100215] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10725.215141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.612469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10729.718554] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10730.542061] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10738.600991] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10740.319737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10746.203970] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10747.991601] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10755.570202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.599847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10755.604575] Lustre: Skipped 2 previous similar messages [10760.679524] LustreError: 323943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.693145] LustreError: 323943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10760.801165] Lustre: server umount lustre-OST0000 complete [10765.806520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10765.819251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10765.823617] Lustre: Skipped 2 previous similar messages [10770.984599] Lustre: server umount lustre-MDT0000 complete [10774.979962] LustreError: 322100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958047 with bad export cookie 13528074887676595462 [10774.995195] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10774.996566] LustreError: 322100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10775.625642] Lustre: server umount lustre-MDT0001 complete [10787.007274] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10789.974082] Key type lgssc unregistered [10790.269211] LNet: 328411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10790.281880] LNetError: 328411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10790.301195] LNet: Removed LNI 192.168.204.105@tcp [10791.119508] Key type .llcrypt unregistered [10791.121205] Key type ._llcrypt unregistered [10814.232923] Key type ._llcrypt registered [10814.235349] Key type .llcrypt registered [10814.326939] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10815.567528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10815.598142] alg: No test for adler32 (adler32-zlib) [10816.650836] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [10816.876757] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10818.639212] Key type lgssc registered [10819.842960] Lustre: Echo OBD driver; http://www.lustre.org/ [10828.174652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10828.188713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10829.592089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10834.031431] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10840.040140] LustreError: 329830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.066644] LustreError: 329830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10843.784822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.107458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10849.228835] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10857.424593] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10863.026597] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10869.734153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10869.739986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10869.747724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10869.762893] Lustre: Skipped 1 previous similar message [10869.778413] Lustre: Skipped 1 previous similar message [10870.818563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10870.842455] Lustre: Skipped 1 previous similar message [10872.569525] Lustre: server umount lustre-MDT0000 complete [10875.889241] LustreError: 329830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10875.937698] LustreError: 329830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10876.639648] LustreError: 329809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958149 with bad export cookie 8831400034408997155 [10876.657760] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10877.244765] Lustre: server umount lustre-MDT0001 complete [10895.031204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10895.307918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10895.358599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10895.606829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10900.969902] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10912.076164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.331113] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10912.369444] Lustre: 332417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10912.734103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10918.164644] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10927.977818] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10933.746752] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10944.833403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.034593] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10945.041960] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10945.049026] LustreError: 334303:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10945.053070] LustreError: 334303:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10945.057772] LustreError: 334303:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10945.116473] Lustre: server umount lustre-OST0000 complete [10945.119271] LustreError: 334303:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10957.493384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10957.867820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10958.182785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10966.179355] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10967.757661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10976.650123] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10985.835033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10985.973603] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10986.094588] Lustre: lustre-OST0001: new disk, initializing [10986.098649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10986.110558] Lustre: 335843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10986.240330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10993.045109] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10995.369538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10995.392990] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10995.551928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11003.364938] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11011.042105] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11011.056022] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11011.057722] LustreError: Skipped 1 previous similar message [11011.078839] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11011.102786] Lustre: Skipped 1 previous similar message [11013.370551] Lustre: server umount lustre-OST0001 complete [11016.167433] LustreError: 334768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.200036] LustreError: 334768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11019.233456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11019.235273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.244372] LustreError: Skipped 1 previous similar message [11019.247311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11019.255373] Lustre: Skipped 2 previous similar messages [11019.268809] Lustre: Skipped 1 previous similar message [11021.292316] LustreError: 336309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.318234] LustreError: 336309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11024.848411] Lustre: server umount lustre-OST0000 complete [11030.496848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11030.502846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11030.516681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11030.522378] Lustre: Skipped 2 previous similar messages [11035.839398] Lustre: server umount lustre-MDT0000 complete [11036.642028] LustreError: 332428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11036.676835] LustreError: 332428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11040.953419] LustreError: 333166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958313 with bad export cookie 8831400034408997708 [11040.957738] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11040.967411] LustreError: 333166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11041.431842] Lustre: server umount lustre-MDT0001 complete [11058.059682] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11062.191803] Key type lgssc unregistered [11062.605603] LNet: 337807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11062.618411] LNetError: 337807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11062.638159] LNet: Removed LNI 192.168.204.105@tcp [11064.023451] Key type .llcrypt unregistered [11064.031520] Key type ._llcrypt unregistered [11138.132588] Key type ._llcrypt registered [11138.135913] Key type .llcrypt registered [11138.250080] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [11153.029585] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11154.175265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11154.304565] alg: No test for adler32 (adler32-zlib) [11155.436461] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [11155.742520] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11157.455330] Key type lgssc registered [11158.630723] Lustre: Echo OBD driver; http://www.lustre.org/ [11197.964880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11197.998508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.291537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11199.355620] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11199.450862] Lustre: lustre-MDT0000: new disk, initializing [11199.542405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11199.559283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11204.435523] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11216.152328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.306471] Lustre: 342746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11216.349924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11216.353771] Lustre: Skipped 1 previous similar message [11216.424499] Lustre: lustre-MDT0001: new disk, initializing [11216.554518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11216.588604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11216.601883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11221.249847] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11231.685994] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11237.987578] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11245.543077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.815452] Lustre: lustre-OST0000: new disk, initializing [11245.819524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11245.828311] Lustre: 344713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11245.906885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11251.594452] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11253.330817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11253.344135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11253.458493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11261.163970] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11265.878237] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11266.149679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11270.170187] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11270.394758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11274.209176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.210507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11274.214309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11274.225850] Lustre: Skipped 1 previous similar message [11278.333449] Lustre: server umount lustre-OST0000 complete [11283.424386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11283.431342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11283.451096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11283.462970] Lustre: Skipped 1 previous similar message [11284.454200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11284.478896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11284.489428] Lustre: Skipped 1 previous similar message [11289.568631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11289.577534] Lustre: Skipped 1 previous similar message [11293.675966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.689270] Lustre: Skipped 1 previous similar message [11296.223124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11296.544210] Lustre: server umount lustre-MDT0000 complete [11300.387813] LustreError: 342738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958573 with bad export cookie 9474553378396898372 [11300.394295] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11300.401958] LustreError: 342738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11300.642745] Lustre: server umount lustre-MDT0001 complete [11310.411116] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:23:01 (1786958581) [11318.022941] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:23:09 (1786958589) [11327.066776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11327.656245] LustreError: 347468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.675958] LustreError: 347468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11327.765976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11333.097957] LustreError: 347469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.667938] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11338.228666] LustreError: 347468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.329906] LustreError: 347469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.424039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.868287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11349.835624] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11358.432377] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11365.126520] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11372.470279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11372.815893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11380.347350] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11391.561373] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11406.023452] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11408.609542] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11410.393549] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11412.178468] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11414.172310] Lustre: Failing over lustre-MDT0000 [11414.410811] Lustre: server umount lustre-MDT0000 complete [11415.520076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11415.533370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11415.573299] LustreError: 348577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.298846] LustreError: 347464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.330136] LustreError: 347464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11433.439906] Lustre: 340969:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786958690/real 1786958690] req@000000005c855221 x1873761720674816/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786958706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11433.479607] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11434.395464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.684038] LustreError: 347465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.707836] LustreError: 347465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11443.680593] LustreError: 340967:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009fe7c2c5 x1873761720679552/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11443.980478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11444.026649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11445.787397] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11448.585647] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11449.353503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11449.413361] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11449.503942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11459.770598] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11461.854946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11466.591731] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11469.511802] Lustre: Failing over lustre-MDT0000 [11469.778894] Lustre: server umount lustre-MDT0000 complete [11469.797779] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.804693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11469.811328] Lustre: Skipped 3 previous similar messages [11474.924534] LustreError: 349961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11474.943911] LustreError: 349961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [11489.588168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.724398] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11489.995757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11490.062426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11491.878560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11495.050942] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11495.403709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11495.416040] Lustre: Skipped 2 previous similar messages [11495.451728] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11495.491899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11506.260771] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11508.180547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11515.874334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.877708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.894669] Lustre: Skipped 2 previous similar messages [11520.860229] Lustre: server umount lustre-MDT0000 complete [11524.994887] LustreError: 347448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958797 with bad export cookie 9474553378396901081 [11525.001540] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.006486] LustreError: 347448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11525.450057] Lustre: server umount lustre-MDT0001 complete [11539.823608] Lustre: server umount lustre-OST0000 complete [11556.634145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.128976] LustreError: 353835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.148062] LustreError: 353835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11557.196595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11561.831728] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11570.971225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.701877] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11585.097335] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11590.584594] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11598.228270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11601.671954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11605.335288] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11613.563130] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11625.958460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11625.975036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.997350] Lustre: Skipped 1 previous similar message [11626.006713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11626.016938] Lustre: Skipped 2 previous similar messages [11629.852165] Lustre: server umount lustre-OST0000 complete [11639.968108] Lustre: server umount lustre-MDT0000 complete [11643.595394] LustreError: 353818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786958916 with bad export cookie 9474553378396901830 [11643.596415] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11643.605769] LustreError: 353818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11643.909554] Lustre: server umount lustre-MDT0001 complete [11655.481223] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11658.143464] Key type lgssc unregistered [11658.398757] LNet: 357371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11658.412317] LNetError: 357371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11658.443910] LNet: Removed LNI 192.168.204.105@tcp [11659.255788] Key type .llcrypt unregistered [11659.264736] Key type ._llcrypt unregistered [11684.598954] Key type ._llcrypt registered [11684.601414] Key type .llcrypt registered [11684.693802] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11685.702755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11685.823616] alg: No test for adler32 (adler32-zlib) [11686.908820] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [11687.202701] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11688.879460] Key type lgssc registered [11690.393233] Lustre: Echo OBD driver; http://www.lustre.org/ [11698.143759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11698.182958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.677402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11705.127143] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11709.921132] LustreError: 358890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11709.955351] LustreError: 358890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11714.069535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.500567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11718.917799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11726.266130] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11731.181464] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11736.827723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11737.215200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11743.084586] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11745.393669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11750.775646] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11763.748340] Lustre: server umount lustre-MDT0000 complete [11765.730935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11765.740126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11765.750743] LustreError: 358890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.246538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11766.272271] Lustre: Skipped 1 previous similar message [11768.368832] LustreError: 358872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959040 with bad export cookie 17475496520097680554 [11768.373860] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11768.386050] LustreError: 358872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11768.803675] Lustre: server umount lustre-MDT0001 complete [11783.942273] Lustre: server umount lustre-OST0000 complete [11798.132301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.567732] LustreError: 362603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11798.580884] LustreError: 362603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11798.645918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.772907] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11804.139529] LustreError: 362604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.259658] LustreError: 362603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.806678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11813.116837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11818.477078] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11828.274432] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11835.337617] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11843.343631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11843.861904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11844.944224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11851.263337] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11860.006654] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11873.781015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11873.796184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11873.806082] Lustre: Skipped 1 previous similar message [11873.824504] Lustre: Skipped 2 previous similar messages [11878.760808] Lustre: server umount lustre-MDT0000 complete [11878.888787] LustreError: 362603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11878.920463] LustreError: 362603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11883.097951] LustreError: 362583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959155 with bad export cookie 17475496520097681681 [11883.120382] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11883.632078] Lustre: server umount lustre-MDT0001 complete [11898.338297] Lustre: server umount lustre-OST0000 complete [11917.270984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.642094] LustreError: 366512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.737100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11923.065634] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11933.009866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.723887] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11947.173604] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11953.058622] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11960.563162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11960.924581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11960.934494] Lustre: Skipped 1 previous similar message [11963.009456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11967.548443] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11976.154534] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11983.843074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11983.852995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.871621] Lustre: Skipped 1 previous similar message [11983.876615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.876464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [11984.885064] LustreError: Skipped 1 previous similar message [11984.893211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.904465] Lustre: Skipped 1 previous similar message [11988.917314] Lustre: server umount lustre-OST0000 complete [11995.106207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11995.109517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11995.113076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11995.118377] Lustre: Skipped 2 previous similar messages [11999.080058] Lustre: server umount lustre-MDT0000 complete [12000.229311] LustreError: 366508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12000.244505] LustreError: 366508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12003.183881] LustreError: 366493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959275 with bad export cookie 17475496520097682703 [12003.186197] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12003.203867] LustreError: 366493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12003.232726] LustreError: 367242:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [12003.245457] LustreError: 367242:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [12003.918490] Lustre: server umount lustre-MDT0001 complete [12016.338505] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12019.441936] Key type lgssc unregistered [12019.723516] LNet: 370050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12019.735176] LNetError: 370050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12019.748518] LNet: Removed LNI 192.168.204.105@tcp [12020.869163] Key type .llcrypt unregistered [12020.879296] Key type ._llcrypt unregistered [12052.473654] Key type ._llcrypt registered [12052.477676] Key type .llcrypt registered [12052.620975] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12053.625634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12053.639589] alg: No test for adler32 (adler32-zlib) [12055.105976] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [12055.580774] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12057.343185] Key type lgssc registered [12058.879710] Lustre: Echo OBD driver; http://www.lustre.org/ [12068.205566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12068.226790] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.238117] LustreError: 371495:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12068.257355] LustreError: 371495:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12068.261615] LustreError: 371495:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12068.271795] LustreError: 371495:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12068.283595] LustreError: 371495:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12077.207683] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12077.214676] LustreError: 371892:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12077.229586] LustreError: 371892:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12077.235321] LustreError: 371892:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12077.244690] LustreError: 371892:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12077.258641] LustreError: 371892:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12097.945394] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12100.818942] Key type lgssc unregistered [12101.145587] LNet: 372898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12101.158245] LNetError: 372898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12101.185867] LNet: Removed LNI 192.168.204.105@tcp [12102.191599] Key type .llcrypt unregistered [12102.195679] Key type ._llcrypt unregistered [12121.933740] Key type ._llcrypt registered [12121.938457] Key type .llcrypt registered [12122.106440] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [12137.276838] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12138.656153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12138.675314] alg: No test for adler32 (adler32-zlib) [12139.867641] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [12140.140803] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12141.815382] Key type lgssc registered [12143.008299] Lustre: Echo OBD driver; http://www.lustre.org/ [12181.526495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12181.604527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.892398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12182.958554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12183.071934] Lustre: lustre-MDT0000: new disk, initializing [12183.200696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12183.243966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12188.639160] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12199.855786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12199.994873] Lustre: 376056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12200.044482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12200.046977] Lustre: Skipped 1 previous similar message [12200.117818] Lustre: lustre-MDT0001: new disk, initializing [12200.244237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12200.325150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12200.352376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12205.047817] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12216.114494] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12223.153375] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12230.516750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12230.802190] Lustre: lustre-OST0000: new disk, initializing [12230.807623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12230.816468] Lustre: 378027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12230.915066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.756213] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12237.394514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12237.402614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12237.517147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12246.629572] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12251.506113] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12251.861353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12256.553456] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12256.762831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12263.407596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12263.413254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12263.422734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12263.442102] Lustre: Skipped 1 previous similar message [12264.702110] Lustre: server umount lustre-OST0000 complete [12272.096475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12272.109028] LustreError: Skipped 1 previous similar message [12272.115783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12272.131861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12272.136843] Lustre: Skipped 1 previous similar message [12273.632600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12273.634400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12273.670409] Lustre: Skipped 1 previous similar message [12278.755738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12278.768375] Lustre: Skipped 1 previous similar message [12283.359177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12283.619789] Lustre: server umount lustre-MDT0000 complete [12287.683199] LustreError: 378026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959560 with bad export cookie 4425922158371789485 [12287.691238] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12287.699744] LustreError: 378026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12288.061273] Lustre: server umount lustre-MDT0001 complete [12298.529201] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12300.916663] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:39:31 (1786959571) [12308.245220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12308.794783] LustreError: 380487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.819119] LustreError: 380487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12308.945068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12314.027626] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12314.084262] LustreError: 380488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.209821] LustreError: 380487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.926754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.363091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12329.720992] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12338.038140] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12343.737525] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12350.678508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12351.069868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12357.360776] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12364.850166] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12372.144137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12372.355981] Lustre: lustre-OST0001: new disk, initializing [12372.360725] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12372.374636] Lustre: 383325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12372.470629] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12378.714485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12378.727769] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12378.767750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12378.812924] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12388.798236] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12394.466826] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12394.476734] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12394.494822] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12394.510153] Lustre: Skipped 1 previous similar message [12398.720974] Lustre: server umount lustre-OST0001 complete [12399.597802] LustreError: 382313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12399.633674] LustreError: 382313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12404.717953] LustreError: 382312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12404.736346] LustreError: 382312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12414.950713] LustreError: 383688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12414.970854] LustreError: 383688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12420.064422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12420.069744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12420.078601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12420.096878] Lustre: Skipped 2 previous similar messages [12420.105902] Lustre: Skipped 2 previous similar messages [12430.303182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12430.560144] Lustre: server umount lustre-OST0000 complete [12435.424689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12435.445372] Lustre: Skipped 1 previous similar message [12440.544660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12440.557165] Lustre: Skipped 8 previous similar messages [12440.878779] Lustre: server umount lustre-MDT0000 complete [12444.597380] LustreError: 381224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959717 with bad export cookie 4425922158371790367 [12444.609864] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12444.616393] LustreError: 381224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12451.128342] Lustre: server umount lustre-MDT0001 complete [12462.884535] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12465.694782] Key type lgssc unregistered [12465.988206] LNet: 385320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12466.009240] LNetError: 385320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12466.034253] LNet: Removed LNI 192.168.204.105@tcp [12466.942320] Key type .llcrypt unregistered [12466.944255] Key type ._llcrypt unregistered [12522.156074] Key type ._llcrypt registered [12522.157943] Key type .llcrypt registered [12522.270524] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12524.118370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12524.183958] alg: No test for adler32 (adler32-zlib) [12525.762044] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [12526.283793] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12528.064320] Key type lgssc registered [12529.190975] Lustre: Echo OBD driver; http://www.lustre.org/ [12538.595270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12538.621817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12539.927787] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12539.964184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12540.213459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12545.684562] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12555.190881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.409955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12555.436856] Lustre: 388136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12555.646879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12560.792202] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12568.810656] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12575.506468] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12581.346783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12581.352637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12581.363631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12582.369853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12582.371375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12582.408029] Lustre: Skipped 1 previous similar message [12584.617906] Lustre: server umount lustre-MDT0000 complete [12588.786232] LustreError: 388127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959861 with bad export cookie 10094897814700480334 [12588.805898] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12588.806229] LustreError: 388127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12589.305490] Lustre: server umount lustre-MDT0001 complete [12601.631168] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:44:32 (1786959872) [12609.322648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.956670] LustreError: 390649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12609.977769] LustreError: 390649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12610.022303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12615.147491] LustreError: 390650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.177974] LustreError: 390650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12615.857689] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12620.259435] LustreError: 390649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12624.925139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.172820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12629.650960] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12638.342807] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12644.636490] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12653.359602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12653.528604] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12653.804085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12660.520296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12668.556772] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12676.149495] Lustre: Permanently deactivating lustre-OST0000 [12676.168897] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12684.900498] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12686.171836] LustreError: 393637:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12686.187077] LustreError: 393637:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12686.196657] LustreError: 393637:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12690.136863] LustreError: 393686:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12690.141477] LustreError: 393686:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12698.582055] Lustre: server umount lustre-OST0000 complete [12701.928725] LustreError: 393937:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12701.935398] LustreError: 393937:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12707.297308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12707.308424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12707.319930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12712.930877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12712.946331] Lustre: Skipped 2 previous similar messages [12714.188494] Lustre: server umount lustre-MDT0000 complete [12718.052547] LustreError: 390649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.080850] LustreError: 390649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12718.945329] LustreError: 390630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786959991 with bad export cookie 10094897814700480915 [12718.963873] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12718.965759] LustreError: 390630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12729.554299] Lustre: server umount lustre-MDT0001 complete [12739.718874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12745.158490] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12748.183913] LustreError: 395097:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12755.205159] Lustre: 395436:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.105@tcp [12757.545426] Lustre: server umount MGS complete [12770.195080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12770.668650] LustreError: 396163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.690367] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12770.711290] Lustre: Skipped 1 previous similar message [12770.775519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12775.502612] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12781.025251] LustreError: 396164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12781.052665] LustreError: 396164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12785.623475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.986786] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12790.579264] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12799.828903] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12806.919797] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12809.143975] Lustre: Permanently reactivating lustre-OST0000 [12809.157472] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12809.161701] Lustre: Skipped 2 previous similar messages [12835.298842] LustreError: 397896:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12844.423125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12844.919421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12844.934611] Lustre: Skipped 1 previous similar message [12851.743359] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12860.527173] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12868.431183] Lustre: server umount lustre-OST0000 complete [12873.185674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12873.200338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12873.222247] Lustre: Skipped 1 previous similar message [12873.240962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12883.042422] Lustre: server umount lustre-MDT0000 complete [12885.988896] LustreError: 396891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.017625] LustreError: 396891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12887.007730] Lustre: 387458:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786960143/real 1786960143] req@000000009332d486 x1873763157247872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786960159 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12887.008126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12887.051392] Lustre: 387458:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12887.072714] Lustre: Skipped 1 previous similar message [12888.007934] LustreError: 396144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960160 with bad export cookie 10094897814700481965 [12888.008399] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12888.016190] LustreError: 396144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12888.506269] Lustre: server umount lustre-MDT0001 complete [12901.407780] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12904.866279] Key type lgssc unregistered [12905.254058] LNet: 400485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12905.278720] LNetError: 400485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12906.367865] LNet: Removed LNI 192.168.204.105@tcp [12907.334517] Key type .llcrypt unregistered [12907.337102] Key type ._llcrypt unregistered [12927.255832] Key type ._llcrypt registered [12927.261243] Key type .llcrypt registered [12927.361429] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [12946.252096] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12947.532113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12947.551570] alg: No test for adler32 (adler32-zlib) [12948.873045] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [12949.154843] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12950.855246] Key type lgssc registered [12952.245727] Lustre: Echo OBD driver; http://www.lustre.org/ [12994.047476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12994.084798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12995.398353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12995.462783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12995.601856] Lustre: lustre-MDT0000: new disk, initializing [12995.712189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12995.730993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13001.303104] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13014.208701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13014.400212] Lustre: 403644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13014.490539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13014.499745] Lustre: Skipped 1 previous similar message [13014.593404] Lustre: lustre-MDT0001: new disk, initializing [13014.745361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13014.822294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13014.833965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13020.616364] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13032.637914] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13040.512889] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13048.836893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13049.135470] Lustre: lustre-OST0000: new disk, initializing [13049.139747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13049.147185] Lustre: 405615:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13049.237956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13055.560458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13055.565494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13055.665640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13055.686190] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13066.448586] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13071.622302] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13071.868450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13077.124470] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13077.426656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13081.572563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.575230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13081.589079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13081.589676] LustreError: Skipped 1 previous similar message [13081.607776] Lustre: Skipped 1 previous similar message [13085.904962] Lustre: server umount lustre-OST0000 complete [13091.830085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13091.836556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13091.845829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13091.846705] Lustre: Skipped 2 previous similar messages [13098.391227] Lustre: server umount lustre-MDT0000 complete [13102.051493] LustreError: 403655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.069804] LustreError: 403655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13102.512950] LustreError: 403636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960375 with bad export cookie 11085024863623169579 [13102.519934] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13102.523643] LustreError: 403636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13102.908812] Lustre: server umount lustre-MDT0001 complete [13113.843137] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13116.352073] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:53:06 (1786960386) [13124.200568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13124.731313] LustreError: 408076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.811641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13129.622271] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13130.209096] LustreError: 408077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.332343] LustreError: 408076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.707615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13140.148554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13145.096978] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13152.984779] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13159.714115] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13167.838756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13168.471672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13175.039816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13183.488207] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13185.404866] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13188.219475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13195.238728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13195.241840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13195.266234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13195.284149] Lustre: Skipped 2 previous similar messages [13200.354546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13200.362565] Lustre: Skipped 1 previous similar message [13200.524356] Lustre: server umount lustre-OST0000 complete [13205.475534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13205.479687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13205.494210] Lustre: Skipped 2 previous similar messages [13210.594165] LustreError: 408076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.675070] Lustre: server umount lustre-MDT0000 complete [13214.280204] LustreError: 408057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960486 with bad export cookie 11085024863623170461 [13214.288336] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13214.296355] LustreError: 408057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13214.573494] Lustre: server umount lustre-MDT0001 complete [13224.624977] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13227.184154] Key type lgssc unregistered [13227.479349] LNet: 411805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13227.487780] LNetError: 411805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13227.506345] LNet: Removed LNI 192.168.204.105@tcp [13228.329718] Key type .llcrypt unregistered [13228.332194] Key type ._llcrypt unregistered [13252.137415] Key type ._llcrypt registered [13252.140475] Key type .llcrypt registered [13252.282321] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13252.690593] Key type .llcrypt unregistered [13252.697144] Key type ._llcrypt unregistered [13274.403458] Key type ._llcrypt registered [13274.414453] Key type .llcrypt registered [13274.642572] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13275.789362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13275.926144] alg: No test for adler32 (adler32-zlib) [13277.269430] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [13277.731770] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13279.407270] Key type lgssc registered [13281.114910] Lustre: Echo OBD driver; http://www.lustre.org/ [13291.164583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13291.212079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13292.702922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13297.890883] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13303.269251] LustreError: 414027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13303.308743] LustreError: 414027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13308.315337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13308.385850] LustreError: 414028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13308.890207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13314.017443] LustreError: 414027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13314.033174] LustreError: 414027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13316.428819] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13318.125456] LustreError: 414023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13318.147514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13318.149245] LustreError: 414023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13323.240931] LustreError: 414028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13323.272644] LustreError: 414028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13326.230110] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13333.493170] LustreError: 414024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13333.521480] LustreError: 414024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13334.046861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13334.639071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13340.271507] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13349.859908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13349.883992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.915617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13351.387766] Lustre: server umount lustre-OST0000 complete [13355.498119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13355.517555] Lustre: Skipped 1 previous similar message [13355.526106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13355.532700] Lustre: Skipped 3 previous similar messages [13357.736292] Lustre: server umount lustre-MDT0000 complete [13360.608220] LustreError: 415003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13360.623375] LustreError: 415003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13361.563261] LustreError: 414009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960634 with bad export cookie 9607480064494090747 [13361.572900] LustreError: 414009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13361.579326] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13361.996173] Lustre: server umount lustre-MDT0001 complete [13373.987843] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13376.841678] Key type lgssc unregistered [13377.253801] LNet: 417179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13377.269828] LNetError: 417179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13377.304227] LNet: Removed LNI 192.168.204.105@tcp [13378.176597] Key type .llcrypt unregistered [13378.178868] Key type ._llcrypt unregistered [13404.656807] Key type ._llcrypt registered [13404.662295] Key type .llcrypt registered [13404.772822] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13406.400236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13406.485063] alg: No test for adler32 (adler32-zlib) [13407.539918] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [13407.756838] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13409.440137] Key type lgssc registered [13410.696542] Lustre: Echo OBD driver; http://www.lustre.org/ [13419.448662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13419.479748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13434.975612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13440.095314] LustreError: 418570:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [13466.784763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13474.096283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13482.161890] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13489.755776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13490.429343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13490.498365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13495.532220] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13495.779746] LustreError: 418595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13495.805450] LustreError: 418595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13500.903474] LustreError: 418596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13501.929097] LustreError: 420209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13501.960956] LustreError: 420209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13504.163277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13504.465700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13508.736771] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13516.789677] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13523.803075] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13531.619317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13531.627487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13531.643571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13533.587221] Lustre: server umount lustre-OST0000 complete [13538.798991] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13538.810338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13538.835119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13539.648365] Lustre: server umount lustre-MDT0000 complete [13540.321835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13540.328774] LustreError: 419613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13540.354653] LustreError: 419613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13543.322148] LustreError: 418577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960815 with bad export cookie 9091512542029208229 [13543.359395] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13552.719409] Lustre: server umount lustre-MDT0001 complete [13564.874570] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13567.842341] Key type lgssc unregistered [13568.163028] LNet: 422143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13568.175287] LNetError: 422143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13568.200994] LNet: Removed LNI 192.168.204.105@tcp [13569.062944] Key type .llcrypt unregistered [13569.065109] Key type ._llcrypt unregistered [13595.668710] Key type ._llcrypt registered [13595.670926] Key type .llcrypt registered [13595.812123] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13596.842133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13596.960634] alg: No test for adler32 (adler32-zlib) [13597.985613] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [13598.216728] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13599.871216] Key type lgssc registered [13600.932799] Lustre: Echo OBD driver; http://www.lustre.org/ [13608.740524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13608.760769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13610.238984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13614.801941] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13619.686608] LustreError: 423574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13619.707017] LustreError: 423574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13624.803288] LustreError: 423575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13625.229133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13625.725862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13630.707847] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13638.603216] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13644.910910] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13652.431309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13652.787543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13654.836399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13659.398523] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13666.993749] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13673.196592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13673.204976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13673.214125] Lustre: Skipped 2 previous similar messages [13673.239896] Lustre: Skipped 2 previous similar messages [13674.602272] Lustre: server umount lustre-MDT0000 complete [13675.060499] LustreError: 424573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13677.297195] LustreError: 425418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.5@tcp arrived at 1786960949 with bad export cookie 1531533356619124143 [13677.324866] LustreError: 425418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13678.572591] LustreError: 424573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13678.609364] LustreError: 424573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13679.313927] LustreError: 423555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786960951 with bad export cookie 1531533356619123254 [13679.325456] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13679.328078] LustreError: 423555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13683.683304] LustreError: 423575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13683.688761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13683.709610] LustreError: 423575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13683.731238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13685.561444] Lustre: server umount lustre-OST0000 complete [13691.698229] LustreError: 424304:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2c86479 x1873764281255040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13691.857587] Lustre: server umount lustre-MDT0001 complete [13700.812942] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13702.853274] Key type lgssc unregistered [13703.068556] LNet: 427264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13703.074431] LNetError: 427264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13703.087633] LNet: Removed LNI 192.168.204.105@tcp [13703.761553] Key type .llcrypt unregistered [13703.764076] Key type ._llcrypt unregistered [13725.942440] Key type ._llcrypt registered [13725.950368] Key type .llcrypt registered [13726.042709] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13726.719690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13726.737693] alg: No test for adler32 (adler32-zlib) [13727.730856] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [13727.979048] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13729.671572] Key type lgssc registered [13730.668968] Lustre: Echo OBD driver; http://www.lustre.org/ [13737.118251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13737.146948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13738.537206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13743.106518] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13748.721483] LustreError: 428681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13748.749270] LustreError: 428681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13753.724579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13753.827708] LustreError: 428680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13754.261611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13759.320080] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13768.206315] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13773.807833] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13779.494354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13779.802205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13783.953667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13787.741770] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13796.874753] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13804.514475] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13804.534440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13804.539924] Lustre: Skipped 1 previous similar message [13804.549424] Lustre: Skipped 1 previous similar message [13805.121992] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [13807.964354] Lustre: server umount lustre-MDT0001 complete [13809.635082] LustreError: 428681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13809.653103] LustreError: 428681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13814.756823] LustreError: 428680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13814.758130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13814.758420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13814.779075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13819.328951] Lustre: server umount lustre-OST0000 complete [13819.875227] LustreError: 429408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.662097] Lustre: server umount lustre-MDT0000 complete [13836.163785] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13838.886367] Key type lgssc unregistered [13839.121401] LNet: 432370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13839.132235] LNetError: 432370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13839.156443] LNet: Removed LNI 192.168.204.105@tcp [13839.959190] Key type .llcrypt unregistered [13839.962107] Key type ._llcrypt unregistered [13865.331919] Key type ._llcrypt registered [13865.333950] Key type .llcrypt registered [13865.429079] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [13881.654892] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13882.760514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13882.888717] alg: No test for adler32 (adler32-zlib) [13884.167385] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [13884.425401] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13886.127197] Key type lgssc registered [13887.179917] Lustre: Echo OBD driver; http://www.lustre.org/ [13926.368728] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13941.291911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13941.330305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13942.543187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13942.574972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13942.677605] Lustre: lustre-MDT0000: new disk, initializing [13942.787248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13942.829081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13947.357606] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13961.667582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13961.884709] Lustre: lustre-OST0000: new disk, initializing [13961.887609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13961.890588] Lustre: Skipped 1 previous similar message [13961.893937] Lustre: 437406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13961.961465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13966.716143] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13968.455528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13968.461882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13968.484107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13974.120488] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13987.258968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13987.357857] Lustre: 436389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13987.416941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13987.469703] Lustre: lustre-MDT0001: new disk, initializing [13987.547077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13987.576750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13987.586906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13991.031639] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13998.435932] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13998.603452] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14002.118721] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14002.320230] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14006.836272] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [14009.825587] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14009.825806] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14009.844477] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14009.863720] Lustre: Skipped 1 previous similar message [14011.934109] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [14012.528972] Lustre: server umount lustre-MDT0001 complete [14020.080254] LustreError: 436401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14020.109738] LustreError: 436401:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14025.215463] LustreError: 439890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14025.239629] LustreError: 439890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14030.309962] LustreError: 436400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14030.362547] LustreError: 436400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14034.914914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14034.919905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14034.931532] Lustre: Skipped 1 previous similar message [14034.936894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14035.425567] LustreError: 436401:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14038.821983] Lustre: server umount lustre-OST0000 complete [14040.544651] LustreError: 436400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14048.844777] Lustre: server umount lustre-MDT0000 complete [14061.792551] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [14064.271797] Key type lgssc unregistered [14064.504131] LNet: 441155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14064.512906] LNetError: 441155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14064.531847] LNet: Removed LNI 192.168.204.105@tcp [14065.269464] Key type .llcrypt unregistered [14065.272762] Key type ._llcrypt unregistered [14106.029315] Key type ._llcrypt registered [14106.032451] Key type .llcrypt registered [14106.149232] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [14106.500409] Key type .llcrypt unregistered [14106.504607] Key type ._llcrypt unregistered [14122.735231] Key type ._llcrypt registered [14122.737966] Key type .llcrypt registered [14122.875950] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [14123.903370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14123.964663] alg: No test for adler32 (adler32-zlib) [14125.031454] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [14125.292293] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [14126.935348] Key type lgssc registered [14127.900752] Lustre: Echo OBD driver; http://www.lustre.org/ [14152.906731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14152.938578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14154.156297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14154.196401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14154.293812] Lustre: lustre-MDT0000: new disk, initializing [14154.351098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14154.362968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14158.510167] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [14168.520078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14168.603807] Lustre: 444550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14168.651985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14168.659732] Lustre: Skipped 1 previous similar message [14168.753242] Lustre: lustre-MDT0001: new disk, initializing [14168.897484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14168.946314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14168.960317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14173.516595] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [14183.276446] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14188.812369] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14195.844615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14196.158248] Lustre: lustre-OST0000: new disk, initializing [14196.165708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14196.174805] Lustre: 446520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14196.330744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14201.979164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14201.991875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14202.080516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14203.603782] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [14214.098446] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14222.836673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14222.843596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14222.850745] Lustre: Skipped 2 previous similar messages [14222.870779] Lustre: Skipped 2 previous similar messages [14227.937041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14227.946782] Lustre: Skipped 2 previous similar messages [14228.681638] Lustre: server umount lustre-MDT0000 complete [14232.921597] LustreError: 444540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786961505 with bad export cookie 11649904864898296183 [14232.926970] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14232.934529] LustreError: 444540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14233.057935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14233.075587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14233.343065] Lustre: server umount lustre-MDT0001 complete [14239.681238] Lustre: server umount lustre-OST0000 complete [14250.957480] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:12:01 (1786961521) [14258.396198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14258.952231] LustreError: 448546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14258.971088] LustreError: 448546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14259.091503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14263.383262] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [14264.290170] LustreError: 448547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14264.303785] LustreError: 448547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14268.385919] LustreError: 448546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14273.003642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14273.509356] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14273.679465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14279.070682] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [14288.373368] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14294.610483] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14316.143839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14316.383924] Lustre: Found index 0 for lustre-OST0000, updating log [14316.584924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14322.780177] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [14329.907160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14330.902159] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14339.047428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14339.055744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14339.091845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14351.327219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14351.489467] Lustre: server umount lustre-OST0000 complete [14368.227294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14368.232334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14368.239377] Lustre: Skipped 2 previous similar messages [14368.252466] Lustre: Skipped 6 previous similar messages [14374.476208] Lustre: server umount lustre-MDT0000 complete [14376.160181] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [14377.442186] LustreError: 448546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14377.455516] LustreError: 448546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14378.564061] LustreError: 448528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786961651 with bad export cookie 11649904864898297576 [14378.567804] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14378.580828] LustreError: 448528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14379.205790] Lustre: server umount lustre-MDT0001 complete [14387.999464] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:14:19 (1786961659) [14440.671948] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [15012.904065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15013.323408] LustreError: 454962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15013.340873] LustreError: 454962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15013.386216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15017.974490] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15018.470830] LustreError: 454963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15022.560894] LustreError: 454962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15025.705768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15025.963341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15030.549224] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15037.753646] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15043.396206] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15062.955744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15063.208421] Lustre: Found index 0 for lustre-OST0000, updating log [15063.510963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15067.360769] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [15068.872429] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15076.766634] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15078.620558] Lustre: server umount lustre-OST0000 complete [15083.556099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15083.578675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15083.585270] Lustre: Skipped 1 previous similar message [15083.612253] Lustre: Skipped 3 previous similar messages [15085.672816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15085.687755] Lustre: Skipped 3 previous similar messages [15088.989543] Lustre: server umount lustre-MDT0000 complete [15089.762091] LustreError: 454957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15089.777494] LustreError: 454957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15091.866810] LustreError: 454942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786962364 with bad export cookie 11649904864898298661 [15091.872543] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15091.890807] LustreError: 454942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15092.562112] Lustre: server umount lustre-MDT0001 complete [15123.117869] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [15126.751318] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [15132.777810] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [15169.829249] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:27:20 (1786962440) [15187.507639] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [15190.295341] Key type lgssc unregistered [15190.593375] LNet: 461579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15190.600877] LNetError: 461579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15316.581347] LNet: Removed LNI 192.168.204.105@tcp [15317.317969] Key type .llcrypt unregistered [15317.320290] Key type ._llcrypt unregistered [15335.229405] Key type ._llcrypt registered [15335.231481] Key type .llcrypt registered [15335.285592] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [15335.986528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15336.110540] alg: No test for adler32 (adler32-zlib) [15337.087866] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [15337.233982] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [15338.863760] Key type lgssc registered [15339.953155] Lustre: Echo OBD driver; http://www.lustre.org/ [15344.212866] LNet: Added LNI 192.168.204.105@tcp100 [8/256/0/180] [15345.198760] LNet: Added LNI 192.168.204.105@tcp101 [8/256/0/180] [15346.189359] LNet: Added LNI 192.168.204.105@tcp102 [8/256/0/180] [15347.205173] LNet: Added LNI 192.168.204.105@tcp103 [8/256/0/180] [15348.011796] LNet: Added LNI 192.168.204.105@tcp104 [8/256/0/180] [15348.782041] LNet: Added LNI 192.168.204.105@tcp105 [8/256/0/180] [15349.537979] LNet: Added LNI 192.168.204.105@tcp106 [8/256/0/180] [15350.355988] LNet: Added LNI 192.168.204.105@tcp107 [8/256/0/180] [15351.084557] LNet: Added LNI 192.168.204.105@tcp108 [8/256/0/180] [15352.042335] LNet: Added LNI 192.168.204.105@tcp109 [8/256/0/180] [15352.818472] LNet: Added LNI 192.168.204.105@tcp110 [8/256/0/180] [15353.712484] LNet: Added LNI 192.168.204.105@tcp111 [8/256/0/180] [15354.508458] LNet: Added LNI 192.168.204.105@tcp112 [8/256/0/180] [15355.435691] LNet: Added LNI 192.168.204.105@tcp113 [8/256/0/180] [15356.241470] LNet: Added LNI 192.168.204.105@tcp114 [8/256/0/180] [15357.235327] LNet: Added LNI 192.168.204.105@tcp115 [8/256/0/180] [15358.126720] LNet: Added LNI 192.168.204.105@tcp116 [8/256/0/180] [15359.043643] LNet: Added LNI 192.168.204.105@tcp117 [8/256/0/180] [15360.189524] LNet: Added LNI 192.168.204.105@tcp118 [8/256/0/180] [15361.189612] LNet: Added LNI 192.168.204.105@tcp119 [8/256/0/180] [15361.979517] LNet: Added LNI 192.168.204.105@tcp120 [8/256/0/180] [15363.099342] LNet: Added LNI 192.168.204.105@tcp121 [8/256/0/180] [15364.407824] LNet: Added LNI 192.168.204.105@tcp122 [8/256/0/180] [15365.375921] LNet: Added LNI 192.168.204.105@tcp123 [8/256/0/180] [15366.095935] LNet: Added LNI 192.168.204.105@tcp124 [8/256/0/180] [15366.939486] LNet: Added LNI 192.168.204.105@tcp125 [8/256/0/180] [15367.969455] LNet: Added LNI 192.168.204.105@tcp126 [8/256/0/180] [15368.981835] LNet: Added LNI 192.168.204.105@tcp127 [8/256/0/180] [15370.071904] LNet: Added LNI 192.168.204.105@tcp128 [8/256/0/180] [15371.102857] LNet: Added LNI 192.168.204.105@tcp129 [8/256/0/180] [15371.900110] LNet: Added LNI 192.168.204.105@tcp130 [8/256/0/180] [15372.902914] LNet: Added LNI 192.168.204.105@tcp131 [8/256/0/180] [15374.101429] LNet: Added LNI 192.168.204.105@tcp132 [8/256/0/180] [15375.202836] LNet: Added LNI 192.168.204.105@tcp133 [8/256/0/180] [15376.129824] LNet: Added LNI 192.168.204.105@tcp134 [8/256/0/180] [15377.084689] LNet: Added LNI 192.168.204.105@tcp135 [8/256/0/180] [15384.532347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15384.639082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15385.800307] Lustre: 464614:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [15385.857742] LNetError: 462072:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15385.891638] LNetError: Protocol error connecting to 192.168.204.105@tcp114 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15385.907538] LNetError: Skipped 7 previous similar messages [15386.008500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15386.040374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15386.132712] Lustre: lustre-MDT0000: new disk, initializing [15386.268035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15386.304759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15386.408279] LNet: 5 local NIs in recovery (showing 5): 192.168.204.105@tcp125, 192.168.204.105@tcp126, 192.168.204.105@tcp124, 192.168.204.105@tcp128, 192.168.204.105@tcp129 [15386.413427] LNetError: 464636:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15386.478334] LNetError: 464636:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15386.494818] LNetError: Protocol error connecting to 192.168.204.105@tcp127 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15386.521061] LNetError: Skipped 2 previous similar messages [15387.489807] LNetError: 462074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15387.516028] LNetError: Protocol error connecting to 192.168.204.105@tcp124 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15391.498859] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15402.956805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15403.153332] Lustre: 464631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15403.212680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15403.219304] Lustre: Skipped 1 previous similar message [15403.333825] Lustre: lustre-MDT0001: new disk, initializing [15403.384411] Lustre: 465433:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [15403.402438] Lustre: 465433:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15403.581308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15403.652938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15403.670822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15408.791449] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15412.268648] Lustre: 465812:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [15412.273592] Lustre: 465812:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15416.102413] LNetError: 462072:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15416.117506] LNetError: Protocol error connecting to 192.168.204.105@tcp121 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15420.071764] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15420.220839] LNetError: 462073:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15420.242570] LNetError: 462073:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15420.257204] LNetError: Protocol error connecting to 192.168.204.105@tcp127 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15420.271141] LNetError: Skipped 2 previous similar messages [15427.095263] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15429.472991] LNetError: 462074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15429.483805] LNetError: 462074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [15429.499427] LNetError: Protocol error connecting to 192.168.204.105@tcp121 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15429.515082] LNetError: Skipped 3 previous similar messages [15434.469066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15434.771957] Lustre: lustre-OST0000: new disk, initializing [15434.775724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15434.783566] Lustre: 466604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15434.869364] Lustre: 466609:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [15434.877452] Lustre: 466609:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15434.925954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15440.397776] Lustre: 466901:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.105@tcp: can't add NID 192.168.204.105@tcp130: rc = -75 [15440.412090] Lustre: 466901:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15440.467171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15440.484188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15440.644405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15441.301598] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15445.857731] LNetError: 464637:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15445.867085] LNetError: 464637:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15445.871178] LNetError: Protocol error connecting to 192.168.204.105@tcp119 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15445.875869] LNetError: Skipped 8 previous similar messages [15451.066520] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15461.347533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15461.349630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15461.365610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15461.381995] LustreError: Skipped 1 previous similar message [15464.258938] Lustre: server umount lustre-OST0000 complete [15470.047895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15470.055098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15470.073719] Lustre: Skipped 1 previous similar message [15470.087914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15470.096150] Lustre: Skipped 1 previous similar message [15471.586925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15471.587462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15471.607814] Lustre: Skipped 1 previous similar message [15473.941696] Lustre: server umount lustre-MDT0000 complete [15477.698211] LustreError: 467556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786962750 with bad export cookie 6787508721022462343 [15477.703671] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15477.717210] LustreError: 467556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15478.178715] Lustre: server umount lustre-MDT0001 complete [15480.675922] LNetError: 464638:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.105 [15480.690783] LNetError: 464638:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [15480.699900] LNetError: Protocol error connecting to 192.168.204.105@tcp121 at host 192.168.204.105:988: is it running a compatible version of Lustre? [15480.709276] LNetError: Skipped 11 previous similar messages [15489.486187] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [15492.420292] Key type lgssc unregistered [15492.819867] LNet: 468521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15492.852398] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15492.861227] LNet: Removed LNI 192.168.204.105@tcp [15493.919479] LNet: Removed LNI 192.168.204.105@tcp100 [15493.926231] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15493.939202] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15493.945746] LNet: Removed LNI 192.168.204.105@tcp101 [15495.007172] LNet: Removed LNI 192.168.204.105@tcp102 [15495.014749] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15495.027986] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15495.040983] LNet: Removed LNI 192.168.204.105@tcp103 [15496.095166] LNet: Removed LNI 192.168.204.105@tcp104 [15497.119326] LNet: Removed LNI 192.168.204.105@tcp105 [15497.125102] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15497.137637] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15497.150284] LNet: Removed LNI 192.168.204.105@tcp106 [15498.207174] LNet: Removed LNI 192.168.204.105@tcp107 [15499.231947] LNet: Removed LNI 192.168.204.105@tcp108 [15500.255163] LNet: Removed LNI 192.168.204.105@tcp109 [15501.279203] LNet: Removed LNI 192.168.204.105@tcp110 [15501.284227] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15501.290034] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15501.304146] LNet: Removed LNI 192.168.204.105@tcp111 [15502.368192] LNet: Removed LNI 192.168.204.105@tcp112 [15503.392832] LNet: Removed LNI 192.168.204.105@tcp113 [15504.415690] LNet: Removed LNI 192.168.204.105@tcp114 [15505.439223] LNet: Removed LNI 192.168.204.105@tcp115 [15506.463156] LNet: Removed LNI 192.168.204.105@tcp116 [15507.487471] LNet: Removed LNI 192.168.204.105@tcp117 [15508.511169] LNet: Removed LNI 192.168.204.105@tcp118 [15509.535179] LNet: Removed LNI 192.168.204.105@tcp119 [15509.537190] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15509.546518] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15509.557362] LNet: Removed LNI 192.168.204.105@tcp120 [15510.623148] LNet: Removed LNI 192.168.204.105@tcp121 [15511.647260] LNet: Removed LNI 192.168.204.105@tcp122 [15512.671437] LNet: Removed LNI 192.168.204.105@tcp123 [15513.695351] LNet: Removed LNI 192.168.204.105@tcp124 [15514.720074] LNet: Removed LNI 192.168.204.105@tcp125 [15515.743671] LNet: Removed LNI 192.168.204.105@tcp126 [15516.768109] LNet: Removed LNI 192.168.204.105@tcp127 [15517.794576] LNet: Removed LNI 192.168.204.105@tcp128 [15518.815356] LNet: Removed LNI 192.168.204.105@tcp129 [15518.821860] LNet: Removed LNI 192.168.204.105@tcp130 [15518.833799] LNet: Removed LNI 192.168.204.105@tcp131 [15518.844735] LNet: Removed LNI 192.168.204.105@tcp132 [15518.860542] LNet: Removed LNI 192.168.204.105@tcp133 [15518.862754] LNet: Removed LNI 192.168.204.105@tcp134 [15518.887787] LNet: Removed LNI 192.168.204.105@tcp135 [15519.688410] Key type .llcrypt unregistered [15519.690474] Key type ._llcrypt unregistered [15540.892789] Key type ._llcrypt registered [15540.898197] Key type .llcrypt registered [15541.001817] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [15555.500724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [15556.710595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15556.852912] alg: No test for adler32 (adler32-zlib) [15557.881582] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [15558.159094] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [15559.831823] Key type lgssc registered [15560.788886] Lustre: Echo OBD driver; http://www.lustre.org/ [15592.606483] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:34:23 (1786962863) [15609.227191] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [15612.280068] Key type lgssc unregistered [15612.588121] LNet: 472614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15612.595900] LNetError: 472614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15612.621983] LNet: Removed LNI 192.168.204.105@tcp [15613.372467] Key type .llcrypt unregistered [15613.376333] Key type ._llcrypt unregistered [15630.585161] Key type ._llcrypt registered [15630.590614] Key type .llcrypt registered [15630.727305] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [15631.813646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15631.910493] alg: No test for adler32 (adler32-zlib) [15633.066324] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [15633.415360] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [15635.119176] Key type lgssc registered [15636.220519] Lustre: Echo OBD driver; http://www.lustre.org/ [15646.691357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15646.724399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15647.967309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15647.986083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15648.113992] Lustre: lustre-MDT0000: new disk, initializing [15648.210203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15648.234387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15653.425791] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15664.833846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15665.003373] Lustre: 473829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15665.085384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15665.091974] Lustre: Skipped 1 previous similar message [15665.181261] Lustre: lustre-MDT0001: new disk, initializing [15665.300549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15665.380484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15665.396273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15670.406901] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15680.314079] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15686.772902] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15694.610572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15694.914708] Lustre: lustre-OST0000: new disk, initializing [15694.920207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15694.930456] Lustre: 475798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15695.038411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15701.572850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15701.588781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15701.677300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15702.336750] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [15713.126928] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15717.022549] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing lnet_if_list [15721.991145] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15722.025148] LNet: Added LNI 192.168.204.105@tcp10 [8/256/0/180] [15730.054500] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15730.084099] LNet: Added LNI 192.168.204.105@tcp11 [8/256/0/180] [15735.064521] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15735.090824] LNet: Added LNI 192.168.204.105@tcp12 [8/256/0/180] [15741.586296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15741.656849] LNet: Added LNI 192.168.204.105@tcp13 [8/256/0/180] [15746.867967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15746.920426] LNet: Added LNI 192.168.204.105@tcp14 [8/256/0/180] [15751.696813] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15751.747906] LNet: Added LNI 192.168.204.105@tcp15 [8/256/0/180] [15757.138650] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15757.168851] LNet: Added LNI 192.168.204.105@tcp16 [8/256/0/180] [15762.956100] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15762.981550] LNet: Added LNI 192.168.204.105@tcp17 [8/256/0/180] [15768.418410] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15768.456425] LNet: Added LNI 192.168.204.105@tcp18 [8/256/0/180] [15774.302673] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15774.344151] LNet: Added LNI 192.168.204.105@tcp19 [8/256/0/180] [15780.209632] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15780.265186] LNet: Added LNI 192.168.204.105@tcp20 [8/256/0/180] [15786.334637] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15786.369421] LNet: Added LNI 192.168.204.105@tcp21 [8/256/0/180] [15792.430701] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15792.479942] LNet: Added LNI 192.168.204.105@tcp22 [8/256/0/180] [15798.537353] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15798.570932] LNet: Added LNI 192.168.204.105@tcp23 [8/256/0/180] [15805.054680] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15805.082604] LNet: Added LNI 192.168.204.105@tcp24 [8/256/0/180] [15810.558601] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15810.609098] LNet: Added LNI 192.168.204.105@tcp25 [8/256/0/180] [15816.592734] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15816.634569] LNet: Added LNI 192.168.204.105@tcp26 [8/256/0/180] [15823.047647] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15823.086761] LNet: Added LNI 192.168.204.105@tcp27 [8/256/0/180] [15828.788824] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15828.821623] LNet: Added LNI 192.168.204.105@tcp28 [8/256/0/180] [15834.624188] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15834.645464] LNet: Added LNI 192.168.204.105@tcp29 [8/256/0/180] [15841.134366] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15841.166061] LNet: Added LNI 192.168.204.105@tcp30 [8/256/0/180] [15847.779328] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15847.809872] LNet: Added LNI 192.168.204.105@tcp31 [8/256/0/180] [15854.655724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15854.703534] LNet: Added LNI 192.168.204.105@tcp32 [8/256/0/180] [15861.635866] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15861.664329] LNet: Added LNI 192.168.204.105@tcp33 [8/256/0/180] [15868.675413] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15868.718251] LNet: Added LNI 192.168.204.105@tcp34 [8/256/0/180] [15876.228136] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15876.279994] LNet: Added LNI 192.168.204.105@tcp35 [8/256/0/180] [15884.119266] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15884.155632] LNet: Added LNI 192.168.204.105@tcp36 [8/256/0/180] [15891.854848] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15891.888242] LNet: Added LNI 192.168.204.105@tcp37 [8/256/0/180] [15900.268576] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15900.306762] LNet: Added LNI 192.168.204.105@tcp38 [8/256/0/180] [15908.103847] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15908.144095] LNet: Added LNI 192.168.204.105@tcp39 [8/256/0/180] [15917.199135] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15917.232547] LNet: Added LNI 192.168.204.105@tcp40 [8/256/0/180] [15925.709137] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15925.739758] LNet: Added LNI 192.168.204.105@tcp41 [8/256/0/180] [15933.322795] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15933.370686] LNet: Added LNI 192.168.204.105@tcp42 [8/256/0/180] [15941.694673] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15941.725214] LNet: Added LNI 192.168.204.105@tcp43 [8/256/0/180] [15949.168603] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15949.194762] LNet: Added LNI 192.168.204.105@tcp44 [8/256/0/180] [15956.378160] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15956.409916] LNet: Added LNI 192.168.204.105@tcp45 [8/256/0/180] [15964.875473] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15964.924143] LNet: Added LNI 192.168.204.105@tcp46 [8/256/0/180] [15973.162157] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15973.180487] LNet: Added LNI 192.168.204.105@tcp47 [8/256/0/180] [15981.814808] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15981.878756] LNet: Added LNI 192.168.204.105@tcp48 [8/256/0/180] [15991.617645] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15991.659890] LNet: Added LNI 192.168.204.105@tcp49 [8/256/0/180] [16002.061439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16002.098672] LNet: Added LNI 192.168.204.105@tcp50 [8/256/0/180] [16023.526255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16023.540155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16023.563378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16024.548396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16024.550061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16024.565854] Lustre: Skipped 1 previous similar message [16024.573806] Lustre: Skipped 1 previous similar message [16027.918719] Lustre: server umount lustre-MDT0000 complete [16031.743351] LustreError: 475797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786963304 with bad export cookie 7379534651410120735 [16031.746692] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16031.757447] LustreError: 475797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16032.053956] Lustre: server umount lustre-MDT0001 complete [16051.167807] Lustre: 473101:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786963307/real 1786963307] req@000000004effc3a1 x1873766416299648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786963323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16051.190608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16051.741963] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [16053.280644] Lustre: 473101:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786963309/real 1786963309] req@0000000054b3a3d3 x1873766416299776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786963325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16056.288130] Lustre: 473099:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786963312/real 1786963312] req@00000000209282c8 x1873766416299904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786963328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16058.339824] Lustre: 473100:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786963314/real 1786963314] req@00000000c94684cc x1873766416300160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786963330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16058.740089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16062.433968] Lustre: 473102:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786963319/real 1786963319] req@00000000afb7c8f5 x1873766416300288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786963335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16077.857177] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x66695faac0fc8a59 [16077.880861] Lustre: MGC192.168.204.105@tcp: Connection restored to 0@lo (at 0@lo) [16078.298776] LustreError: 475790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16078.334345] LustreError: 475790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16078.376631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16079.394467] LustreError: 475791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16079.397588] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16080.422344] LustreError: 475790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16083.009308] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16085.476699] LustreError: 475791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16090.599821] LustreError: 475790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16092.666516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16093.117760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16096.184211] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16098.026512] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16105.948654] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16112.601740] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16137.196570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16137.200884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16137.217381] Lustre: Skipped 1 previous similar message [16137.231538] Lustre: Skipped 2 previous similar messages [16141.715473] Lustre: server umount lustre-OST0000 complete [16147.424633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16147.427598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16147.432150] Lustre: Skipped 1 previous similar message [16147.448198] Lustre: Skipped 1 previous similar message [16151.728667] Lustre: server umount lustre-MDT0000 complete [16152.546530] LustreError: 501480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16152.563055] LustreError: 501480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16155.666769] LustreError: 473822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786963428 with bad export cookie 7379534651410123353 [16155.681764] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16156.014461] Lustre: server umount lustre-MDT0001 complete [16167.390800] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [16170.070639] Key type lgssc unregistered [16170.534982] LNet: 502845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16170.549634] LNetError: 502845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16171.615638] LNet: Removed LNI 192.168.204.105@tcp [16171.617621] LNetError: 502845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16171.629648] LNet: Removed LNI 192.168.204.105@tcp10 [16171.632305] LNet: Removed LNI 192.168.204.105@tcp11 [16171.634202] LNet: Removed LNI 192.168.204.105@tcp12 [16171.636603] LNet: Removed LNI 192.168.204.105@tcp13 [16171.638555] LNet: Removed LNI 192.168.204.105@tcp14 [16171.641341] LNet: Removed LNI 192.168.204.105@tcp15 [16171.646850] LNet: Removed LNI 192.168.204.105@tcp16 [16171.652703] LNet: Removed LNI 192.168.204.105@tcp17 [16171.655336] LNet: Removed LNI 192.168.204.105@tcp18 [16171.659582] LNet: Removed LNI 192.168.204.105@tcp19 [16171.662764] LNet: Removed LNI 192.168.204.105@tcp20 [16171.665662] LNet: Removed LNI 192.168.204.105@tcp21 [16171.668152] LNet: Removed LNI 192.168.204.105@tcp22 [16171.671137] LNet: Removed LNI 192.168.204.105@tcp23 [16171.673386] LNet: Removed LNI 192.168.204.105@tcp24 [16171.675568] LNet: Removed LNI 192.168.204.105@tcp25 [16171.677733] LNet: Removed LNI 192.168.204.105@tcp26 [16171.680093] LNet: Removed LNI 192.168.204.105@tcp27 [16171.682439] LNet: Removed LNI 192.168.204.105@tcp28 [16171.684611] LNet: Removed LNI 192.168.204.105@tcp29 [16171.686742] LNet: Removed LNI 192.168.204.105@tcp30 [16171.688949] LNet: Removed LNI 192.168.204.105@tcp31 [16171.690478] LNet: Removed LNI 192.168.204.105@tcp32 [16171.692383] LNet: Removed LNI 192.168.204.105@tcp33 [16171.695347] LNet: Removed LNI 192.168.204.105@tcp34 [16171.698081] LNet: Removed LNI 192.168.204.105@tcp35 [16171.700541] LNet: Removed LNI 192.168.204.105@tcp36 [16171.703080] LNet: Removed LNI 192.168.204.105@tcp37 [16171.705951] LNet: Removed LNI 192.168.204.105@tcp38 [16171.708145] LNet: Removed LNI 192.168.204.105@tcp39 [16171.711339] LNet: Removed LNI 192.168.204.105@tcp40 [16171.713992] LNet: Removed LNI 192.168.204.105@tcp41 [16171.719730] LNet: Removed LNI 192.168.204.105@tcp42 [16171.723696] LNet: Removed LNI 192.168.204.105@tcp43 [16171.727737] LNet: Removed LNI 192.168.204.105@tcp44 [16171.732446] LNet: Removed LNI 192.168.204.105@tcp45 [16171.735927] LNet: Removed LNI 192.168.204.105@tcp46 [16171.737898] LNet: Removed LNI 192.168.204.105@tcp47 [16171.740668] LNet: Removed LNI 192.168.204.105@tcp48 [16171.743166] LNet: Removed LNI 192.168.204.105@tcp49 [16171.766715] LNet: Removed LNI 192.168.204.105@tcp50 [16172.855699] Key type .llcrypt unregistered [16172.860325] Key type ._llcrypt unregistered [16193.294879] Key type ._llcrypt registered [16193.297107] Key type .llcrypt registered [16193.418438] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [16210.663183] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [16211.912366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16211.999092] alg: No test for adler32 (adler32-zlib) [16213.131923] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [16213.446174] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [16215.183247] Key type lgssc registered [16216.489582] Lustre: Echo OBD driver; http://www.lustre.org/ [16258.712701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16258.741371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16259.983494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16260.009657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16260.094658] Lustre: lustre-MDT0000: new disk, initializing [16260.185903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16260.214642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16265.133447] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16276.597646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16276.754288] Lustre: 506045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16276.835934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16276.838810] Lustre: Skipped 1 previous similar message [16276.923162] Lustre: lustre-MDT0001: new disk, initializing [16277.043143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16277.086685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16277.107487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16281.842723] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16292.705844] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16299.325453] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16307.649470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16307.818594] Lustre: *** cfs_fail_loc=90f, val=0*** [16307.904744] Lustre: lustre-OST0000: new disk, initializing [16307.908429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16307.921798] Lustre: 508061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16308.014951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16314.981765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16314.991626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16315.113177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16315.152353] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16326.349435] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16335.842561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16335.857203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16335.875232] Lustre: Skipped 1 previous similar message [16335.876371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16335.895398] Lustre: Skipped 1 previous similar message [16340.961537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16340.979056] Lustre: Skipped 1 previous similar message [16346.086657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16346.095856] Lustre: Skipped 1 previous similar message [16349.153199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16349.446475] Lustre: server umount lustre-OST0000 complete [16353.764068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16353.782674] LustreError: Skipped 1 previous similar message [16353.795941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16353.818799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16356.335974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16359.647329] Lustre: server umount lustre-MDT0000 complete [16363.330758] LustreError: 506037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786963635 with bad export cookie 7610390302735397702 [16363.339833] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16363.596286] Lustre: server umount lustre-MDT0001 complete [16376.533496] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [16379.820314] Key type lgssc unregistered [16380.155335] LNet: 509975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16380.162916] LNetError: 509975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16380.195796] LNet: Removed LNI 192.168.204.105@tcp [16381.307030] Key type .llcrypt unregistered [16381.313610] Key type ._llcrypt unregistered [16403.761464] Key type ._llcrypt registered [16403.763457] Key type .llcrypt registered [16403.860362] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [16404.615918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16404.682502] alg: No test for adler32 (adler32-zlib) [16405.879709] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [16406.332948] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [16408.079202] Key type lgssc registered [16409.617137] Lustre: Echo OBD driver; http://www.lustre.org/ [16420.978234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16420.994996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16422.451975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16427.645769] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16432.610287] LustreError: 511247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16432.651412] LustreError: 511247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16436.930630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16437.372750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16442.048492] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16451.279356] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16458.396993] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16465.768247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16466.250125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16473.369805] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16480.928290] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16491.015310] loop0: detected capacity change from 0 to 204800000 [16491.187644] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16491.198686] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16491.215081] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16498.069625] loop0: detected capacity change from 0 to 204800000 [16498.311379] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16498.331225] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16498.344180] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16504.491434] loop0: detected capacity change from 0 to 204800000 [16511.423493] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16511.690299] Lustre: fs15246-OST0000: new disk, initializing [16511.706648] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16511.725440] Lustre: 514838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16511.840796] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16518.939477] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16530.092186] loop1: detected capacity change from 0 to 204800000 [16537.060239] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16537.171824] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16537.239421] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16537.326347] Lustre: fs15246-MDT0000: new disk, initializing [16537.531505] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16537.557841] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16537.690824] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16538.727857] Lustre: 515963:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004326421e x1873767225671168/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786963827 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16543.348689] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16547.887797] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16547.909969] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16547.936945] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16569.841205] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16569.863098] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16571.356548] Lustre: server umount fs15246-MDT0000 complete [16574.972922] LustreError: 512480:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16580.076035] LustreError: 512480:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16581.712778] Lustre: server umount fs15246-OST0000 complete [16595.428602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16595.460416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16595.467897] Lustre: Skipped 1 previous similar message [16600.547547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16600.559666] Lustre: Skipped 1 previous similar message [16601.713881] Lustre: server umount lustre-OST0000 complete [16610.785318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16610.786486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16610.802063] Lustre: Skipped 2 previous similar messages [16610.823901] Lustre: Skipped 1 previous similar message [16612.900646] Lustre: server umount lustre-MDT0000 complete [16615.904883] LustreError: 515314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16615.921129] LustreError: 515314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16617.153273] LustreError: 511230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786963889 with bad export cookie 9911678800475532912 [16617.164251] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16617.164779] LustreError: 511230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16617.599241] Lustre: server umount lustre-MDT0001 complete [16631.694294] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [16635.016717] Key type lgssc unregistered [16635.314840] LNet: 519063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16635.323582] LNetError: 519063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16635.338512] LNet: Removed LNI 192.168.204.105@tcp [16636.182912] Key type .llcrypt unregistered [16636.186369] Key type ._llcrypt unregistered [16657.544431] Key type ._llcrypt registered [16657.546751] Key type .llcrypt registered [16657.683712] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [16658.828522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16658.850693] alg: No test for adler32 (adler32-zlib) [16659.931853] Lustre: Lustre: Build Version: 2.17.57_2_g91a1b42 [16660.198737] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [16661.881332] Key type lgssc registered [16662.935662] Lustre: Echo OBD driver; http://www.lustre.org/ [16712.009973] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [16724.174717] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [16762.351892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16762.406336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16763.822688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16763.851885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16763.982861] Lustre: lustre-MDT0000: new disk, initializing [16764.193064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16764.210848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16769.640823] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16782.319292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16782.488598] Lustre: 523545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16782.563328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16782.567836] Lustre: Skipped 1 previous similar message [16782.662434] Lustre: lustre-MDT0001: new disk, initializing [16782.844366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16782.919376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16782.937244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16788.452269] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16799.216551] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16805.069885] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16813.094131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16813.461363] Lustre: lustre-OST0000: new disk, initializing [16813.468874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16813.480282] Lustre: 525521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16813.586438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16821.106972] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [16822.365212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16822.390164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16822.608363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16833.894923] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16839.354136] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16839.651330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16845.094316] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16845.455896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16848.357959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16848.362289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16848.403181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16848.405642] Lustre: Skipped 1 previous similar message [16848.425612] Lustre: Skipped 1 previous similar message [16853.476765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16853.486329] Lustre: Skipped 1 previous similar message [16854.198466] Lustre: server umount lustre-OST0000 complete [16858.594663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16858.599894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16858.607023] Lustre: Skipped 1 previous similar message [16858.618684] Lustre: Skipped 1 previous similar message [16863.715732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16863.726554] Lustre: Skipped 1 previous similar message [16864.178770] Lustre: server umount lustre-MDT0000 complete [16868.307805] LustreError: 523537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786964140 with bad export cookie 5794126872076925809 [16868.314803] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16868.329493] LustreError: 523537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16868.769304] Lustre: server umount lustre-MDT0001 complete [16897.242361] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [16912.836099] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [16954.623976] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16610 sec ========= 06:57:05 (1786964225) [16956.608872] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:57:07 (1786964227) === [16961.792801] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:57:11 (1786964231) === [17007.263054] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [17010.841919] Key type lgssc unregistered [17011.137264] LNet: 532191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17011.146836] LNetError: 532191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17011.163669] LNet: Removed LNI 192.168.204.105@tcp [17012.076511] Key type .llcrypt unregistered [17012.083108] Key type ._llcrypt unregistered