[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474184889 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003176] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008608] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011151] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.013050] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022127] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027770] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.032341] *** VALIDATE proc *** [ 0.033262] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037110] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040015] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.043775] debug: unmapping init [mem 0xffffffffa2a59000-0xffffffffa2a60fff] [ 0.045172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054294] rcu: Hierarchical SRCU implementation. [ 0.056453] smp: Bringing up secondary CPUs ... [ 0.057603] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.062037] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140523] node 0 deferred pages initialised in 72ms [ 0.145100] devtmpfs: initialized [ 0.146282] x86/mm: Memory block size: 128MB [ 0.149740] gcov: version magic: 0x41383552 [ 0.152161] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156302] pinctrl core: initialized pinctrl subsystem [ 0.158149] [ 0.158746] ************************************************************* [ 0.161013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163009] ** ** [ 0.165011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.168011] ** This means that this kernel is built to expose internal ** [ 0.170011] ** IOMMU data structures, which may compromise security on ** [ 0.172009] ** your system. ** [ 0.174014] ** ** [ 0.176010] ** If you see this message and you are not debugging the ** [ 0.178010] ** kernel, report this immediately to your vendor! ** [ 0.180010] ** ** [ 0.182011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183007] ************************************************************* [ 0.185628] NET: Registered protocol family 16 [ 0.187442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195089] cpuidle: using governor menu [ 0.196906] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199522] PCI: Using configuration type 1 for base access [ 0.201139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216077] cryptd: max_cpu_qlen set to 1000 [ 0.219128] ACPI: Added _OSI(Module Device) [ 0.220012] ACPI: Added _OSI(Processor Device) [ 0.221009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.222026] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226804] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232536] ACPI: Interpreter enabled [ 0.234065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236015] ACPI: Using IOAPIC for interrupt routing [ 0.238104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.241371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.252693] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266362] acpiphp: Slot [2] registered [ 0.267091] acpiphp: Slot [5] registered [ 0.268083] acpiphp: Slot [6] registered [ 0.269082] acpiphp: Slot [7] registered [ 0.270085] acpiphp: Slot [8] registered [ 0.271058] acpiphp: Slot [9] registered [ 0.273115] acpiphp: Slot [10] registered [ 0.274108] acpiphp: Slot [3] registered [ 0.275102] acpiphp: Slot [4] registered [ 0.277100] acpiphp: Slot [11] registered [ 0.278091] acpiphp: Slot [12] registered [ 0.280111] acpiphp: Slot [13] registered [ 0.281097] acpiphp: Slot [14] registered [ 0.282091] acpiphp: Slot [15] registered [ 0.284108] acpiphp: Slot [16] registered [ 0.285118] acpiphp: Slot [17] registered [ 0.286090] acpiphp: Slot [18] registered [ 0.288087] acpiphp: Slot [19] registered [ 0.289097] acpiphp: Slot [20] registered [ 0.290128] acpiphp: Slot [21] registered [ 0.292138] acpiphp: Slot [22] registered [ 0.293093] acpiphp: Slot [23] registered [ 0.294131] acpiphp: Slot [24] registered [ 0.296094] acpiphp: Slot [25] registered [ 0.297086] acpiphp: Slot [26] registered [ 0.298096] acpiphp: Slot [27] registered [ 0.299089] acpiphp: Slot [28] registered [ 0.301095] acpiphp: Slot [29] registered [ 0.302118] acpiphp: Slot [30] registered [ 0.303094] acpiphp: Slot [31] registered [ 0.304060] PCI host bridge to bus 0000:00 [ 0.306023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324109] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.342021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.344042] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.346023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.349614] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.352860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386503] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439910] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.497615] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.510015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.525026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.534584] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.566016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.573981] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.583013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615586] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.640018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.651017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.664024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.675870] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.678447] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.681541] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.684417] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.687305] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.692032] iommu: Default domain type: Passthrough [ 0.694404] SCSI subsystem initialized [ 0.696216] ACPI: bus type USB registered [ 0.697182] usbcore: registered new interface driver usbfs [ 0.700148] usbcore: registered new interface driver hub [ 0.702079] usbcore: registered new device driver usb [ 0.704195] pps_core: LinuxPPS API ver. 1 registered [ 0.706014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.710061] PTP clock support registered [ 0.712099] EDAC MC: Ver: 3.0.0 [ 0.714091] PCI: Using ACPI for IRQ routing [ 0.716887] NetLabel: Initializing [ 0.718009] NetLabel: domain hash size = 128 [ 0.719011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.722105] NetLabel: unlabeled traffic allowed by default [ 0.724187] vgaarb: loaded [ 0.726281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.728015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.732321] clocksource: Switched to clocksource kvm-clock [ 0.846559] VFS: Disk quotas dquot_6.6.0 [ 0.848564] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.852204] *** VALIDATE ramfs *** [ 0.853687] *** VALIDATE hugetlbfs *** [ 0.855505] pnp: PnP ACPI init [ 0.858197] pnp: PnP ACPI: found 6 devices [ 0.874286] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.878104] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.880593] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.882810] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.885492] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.888370] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891693] NET: Registered protocol family 2 [ 0.894471] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.899649] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.903791] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910718] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914721] TCP: Hash tables configured (established 65536 bind 65536) [ 0.918397] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922180] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925601] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928844] NET: Registered protocol family 1 [ 0.932067] RPC: Registered named UNIX socket transport module. [ 0.934494] RPC: Registered udp transport module. [ 0.936262] RPC: Registered tcp transport module. [ 0.938032] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.940821] NET: Registered protocol family 44 [ 0.942844] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.945332] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.947744] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.950448] PCI: CLS 0 bytes, default 64 [ 0.952351] Unpacking initramfs... [ 2.358457] debug: unmapping init [mem 0xffff963f3cc54000-0xffff963f3ffbffff] [ 2.362218] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.364320] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.366879] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.872948] Initialise system trusted keyrings [ 2.874857] Key type blacklist registered [ 2.878848] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887969] zbud: loaded [ 2.891256] *** VALIDATE nfs *** [ 2.892552] *** VALIDATE nfs4 *** [ 2.894654] pstore: using deflate compression [ 2.898188] Platform Keyring initialized [ 2.995449] NET: Registered protocol family 38 [ 2.997529] Key type asymmetric registered [ 2.999274] Asymmetric key parser 'x509' registered [ 3.001368] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.004691] io scheduler mq-deadline registered [ 3.006464] io scheduler kyber registered [ 3.008339] io scheduler bfq registered [ 3.010253] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.013622] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.016902] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.020061] ACPI: Power Button [PWRF] [ 3.024716] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.031241] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.041776] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.047978] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.061787] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.090311] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.120340] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.125492] Non-volatile memory driver v1.3 [ 3.127391] Linux agpgart interface v0.103 [ 3.158650] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.161899] vda: detected capacity change from 0 to 74645504 [ 3.177947] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.184106] vdb: detected capacity change from 0 to 1073741824 [ 3.200393] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.203400] vdc: detected capacity change from 0 to 2621440000 [ 3.227524] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.231312] vdd: detected capacity change from 0 to 2621440000 [ 3.249167] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.251604] vde: detected capacity change from 0 to 4294967296 [ 3.267963] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.271669] vdf: detected capacity change from 0 to 4294967296 [ 3.282371] libphy: Fixed MDIO Bus: probed [ 3.287725] usbcore: registered new interface driver usbserial_generic [ 3.290710] usbserial: USB Serial support registered for generic [ 3.293443] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.298285] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.300501] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.303154] mousedev: PS/2 mouse device common for all mice [ 3.306763] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.307327] rtc_cmos 00:05: RTC can wake from S4 [ 3.315915] rtc_cmos 00:05: registered as rtc0 [ 3.317686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.319497] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.327941] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.329689] intel_pstate: CPU model not supported [ 3.336785] hid: raw HID events driver (C) Jiri Kosina [ 3.339133] usbcore: registered new interface driver usbhid [ 3.341285] usbhid: USB HID core driver [ 3.343145] drop_monitor: Initializing network drop monitor service [ 3.346225] Initializing XFRM netlink socket [ 3.348776] NET: Registered protocol family 10 [ 3.352543] Segment Routing with IPv6 [ 3.354276] NET: Registered protocol family 17 [ 3.356666] mpls_gso: MPLS GSO support [ 3.365121] RAS: Correctable Errors collector initialized. [ 3.367712] AVX version of gcm_enc/dec engaged. [ 3.369698] AES CTR mode by8 optimization enabled [ 3.451029] sched_clock: Marking stable (3451017937, 0)->(4343046259, -892028322) [ 3.454196] registered taskstats version 1 [ 3.456547] Loading compiled-in X.509 certificates [ 3.458790] zswap: loaded using pool lzo/zbud [ 3.485784] Key type big_key registered [ 3.504373] Key type encrypted registered [ 3.506458] ima: No TPM chip found, activating TPM-bypass! [ 3.509108] ima: Allocated hash algorithm: sha1 [ 3.511144] ima: No architecture policies found [ 3.512911] evm: Initialising EVM extended attributes: [ 3.515262] evm: security.selinux [ 3.516789] evm: security.ima [ 3.518198] evm: security.capability [ 3.519711] evm: HMAC attrs: 0x1 [ 3.525972] rtc_cmos 00:05: setting system clock to 2026-07-16 07:08:50 UTC (1784185730) [ 3.532965] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.536357] debug: unmapping init [mem 0xffffffffa2782000-0xffffffffa2a58fff] [ 3.547127] Write protecting the kernel read-only data: 28672k [ 3.550543] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 3.554764] debug: unmapping init [mem 0xffffffffa1714000-0xffffffffa17fffff] [ 3.592667] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.602254] systemd[1]: Detected virtualization kvm. [ 3.604648] systemd[1]: Detected architecture x86-64. [ 3.606945] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.635218] systemd[1]: No hostname configured. [ 3.637119] systemd[1]: Set hostname to . [ 3.639710] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642478] systemd[1]: Initializing machine ID from random generator. [ 3.685493] random: ln: uninitialized urandom read (6 bytes read) [ 3.791293] random: systemd: uninitialized urandom read (16 bytes read) [ 3.793968] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.798855] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.803272] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.485539] device-mapper: uevent: version 1.0.3 [ 4.488188] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 4.863762] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.343167] virtio_net virtio0 ens2: renamed from eth0 [ 5.388979] scsi host0: ata_piix [ 5.472952] scsi host1: ata_piix [ 5.474561] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.477750] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.378159] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.037460] random: crng init done [ 10.039308] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.648182] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.781364] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.051322] SELinux: Disabled at runtime. [ 12.122452] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.131779] systemd[1]: Detected virtualization kvm. [ 12.133635] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.631909] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.635704] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.640668] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.645011] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.648432] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.656437] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.665234] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK [[ 12.773776] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.131293] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.505687] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.543062] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.657266] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.671073] EDAC sbridge: Ver: 1.1.2 [ 15.302560] Key type dns_resolver registered [ 15.623613] NFS: Registering the id_resolver key type [ 15.625221] Key type id_resolver registered [ 15.627630] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 40.148640] hrtimer: interrupt took 4739910 ns [ 68.015327] libcfs: loading out-of-tree module taints kernel. [ 68.050875] Key type ._llcrypt registered [ 68.056791] Key type .llcrypt registered [ 68.155660] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 89.630786] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 91.443821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.473845] alg: No test for adler32 (adler32-zlib) [ 92.998815] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 94.183348] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 96.072375] Key type lgssc registered [ 98.353863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.716543] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.099278] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 187.744783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 187.772174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.055731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.075182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.167803] Lustre: lustre-MDT0000: new disk, initializing [ 189.247176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.259082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.409893] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.640069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.807295] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.862571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 208.869553] Lustre: Skipped 1 previous similar message [ 209.063924] Lustre: lustre-MDT0001: new disk, initializing [ 209.159901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.200230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.219638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.000639] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.790889] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.863374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.402172] Lustre: lustre-OST0000: new disk, initializing [ 231.413893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.426841] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.500864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.810096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.842867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.042461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 239.022635] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.588394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.764335] Lustre: lustre-OST0001: new disk, initializing [ 260.767571] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.772599] Lustre: 9529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.844286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.599839] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 268.399254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.413811] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.499658] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 280.499754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.133947] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.338468] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 303.267552] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 309.629416] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 312.846984] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 316.385560] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 319.076523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 16 03:14:03 EDT 2026 [ 339.080675] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 340.766605] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 357.855738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.868394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.883922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.929298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.929581] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.942028] Lustre: Skipped 2 previous similar messages [ 360.962691] Lustre: Skipped 2 previous similar messages [ 363.803842] Lustre: server umount lustre-MDT0000 complete [ 367.800856] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186094 with bad export cookie 13686386547161056661 [ 367.806979] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.820984] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 371.167829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.169212] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.177073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.177083] Lustre: Skipped 1 previous similar message [ 371.182819] Lustre: Skipped 1 previous similar message [ 371.219847] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 372.192418] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.199140] LustreError: 6522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 374.281714] Lustre: server umount lustre-MDT0001 complete [ 378.914302] Lustre: server umount lustre-OST0000 complete [ 383.095855] Lustre: server umount lustre-OST0001 complete [ 391.175662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 402.647758] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 443.391673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.730542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.768218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.862269] Lustre: lustre-MDT0000: new disk, initializing [ 443.933643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.951742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.550821] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 459.357768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.510443] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.602587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.611735] Lustre: Skipped 1 previous similar message [ 459.742515] Lustre: lustre-MDT0001: new disk, initializing [ 459.917963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.979089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 459.999954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 465.011419] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 475.178829] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 482.432760] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.473391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.714205] Lustre: lustre-OST0000: new disk, initializing [ 490.717838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.727896] Lustre: 17055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.794859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.788104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 491.804970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 491.959922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 497.913973] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 509.061704] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 514.051302] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 514.321402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 519.719588] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 520.037738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 522.723674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.727171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.748521] LustreError: Skipped 1 previous similar message [ 522.750169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.750175] Lustre: Skipped 3 previous similar messages [ 522.780214] Lustre: Skipped 1 previous similar message [ 527.843455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.860849] Lustre: Skipped 1 previous similar message [ 528.613524] Lustre: server umount lustre-OST0000 complete [ 532.962918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.982753] Lustre: Skipped 1 previous similar message [ 538.081671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 538.091359] Lustre: Skipped 4 previous similar messages [ 547.295120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.497629] Lustre: server umount lustre-MDT0000 complete [ 548.320736] LustreError: 16506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.337237] LustreError: 16506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 551.204390] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186278 with bad export cookie 13686386547161058194 [ 551.206844] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 551.220844] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 551.575925] Lustre: server umount lustre-MDT0001 complete [ 556.463966] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:18:01 (1784186281) [ 559.207420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 560.916043] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:18:05 (1784186285) [ 568.187859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.840929] LustreError: 19465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.910456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.573670] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 573.922572] LustreError: 19466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.266388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.050778] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 599.135977] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 606.255671] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 614.449674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 622.408211] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 632.563974] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 653.793502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 653.803135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.821565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 653.825918] Lustre: Skipped 2 previous similar messages [ 658.678981] Lustre: server umount lustre-OST0000 complete [ 678.369790] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 678.536850] Lustre: server umount lustre-MDT0000 complete [ 679.397910] LustreError: 20621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.415895] LustreError: 20621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 682.562105] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186409 with bad export cookie 13686386547161059076 [ 682.562641] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.567138] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 682.915612] Lustre: server umount lustre-MDT0001 complete [ 695.817337] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 698.835832] Key type lgssc unregistered [ 699.101690] LNet: 23023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.119731] LNetError: 23023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 699.139418] LNet: Removed LNI 192.168.204.145@tcp [ 700.089241] Key type .llcrypt unregistered [ 700.092739] Key type ._llcrypt unregistered [ 725.940876] Key type ._llcrypt registered [ 725.943399] Key type .llcrypt registered [ 726.066351] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 726.868843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.925911] alg: No test for adler32 (adler32-zlib) [ 727.950498] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 728.166414] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 729.823855] Key type lgssc registered [ 731.038923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 740.247204] loop: module loaded [ 740.316338] loop0: detected capacity change from 0 to 204800000 [ 740.487478] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.517163] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.866385] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.636337] loop0: detected capacity change from 0 to 204800000 [ 747.838739] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.850653] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.868834] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.456861] loop0: detected capacity change from 0 to 204800000 [ 759.098521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.129227] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.379158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 760.490737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 760.592243] Lustre: lustre-MDT0000: new disk, initializing [ 760.663290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.690969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.071043] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 773.312294] loop1: detected capacity change from 0 to 204800000 [ 778.355920] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 778.588177] Lustre: lustre-OST07c6: new disk, initializing [ 778.591241] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 778.595500] Lustre: Skipped 1 previous similar message [ 778.600844] Lustre: 26208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 778.668929] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 784.877561] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 785.560889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 785.580599] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 785.608489] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 791.113616] Lustre: 25198:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 791.121241] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 791.126567] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 791.132420] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 791.138384] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 791.145566] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 806.964833] Lustre: 26882:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 806.975235] Lustre: 26882:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 806.984637] Lustre: 26882:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 806.989953] Lustre: 26882:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 807.000298] Lustre: 26882:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 807.009108] Lustre: 26882:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.570135] Lustre: 26882:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 812.581300] Lustre: 26882:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 812.593520] Lustre: 26882:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 812.600822] Lustre: 26882:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.608106] Lustre: 26882:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 812.616872] Lustre: 26882:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.622166] Lustre: 26882:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 812.629806] Lustre: 26882:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.637372] Lustre: 26882:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 812.646214] Lustre: 26882:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 812.654952] Lustre: 26882:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 812.658863] Lustre: 26882:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.976140] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 815.983024] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 815.988635] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 815.993751] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 815.999049] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 816.006475] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.014703] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 816.020142] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.024510] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 816.028700] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.032819] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.037110] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.319302] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 821.335335] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 821.341641] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.350353] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.359184] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.373665] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.387460] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 821.407165] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.411662] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 821.419540] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.424746] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.429655] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 830.020882] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 830.028248] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 830.033914] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 830.040159] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.048292] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 830.054356] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.069304] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 830.076890] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.087789] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 830.099318] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.112973] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 830.132984] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 846.610690] Lustre: 25198:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 846.623716] Lustre: 25198:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 846.635731] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 846.645716] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 846.655904] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 846.662226] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 846.673418] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 846.684858] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 846.695102] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 846.703143] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 846.714102] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 846.731858] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 883.168600] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.193215] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 888.288241] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 888.697650] Lustre: server umount lustre-OST07c6 complete [ 901.682332] Lustre: server umount lustre-MDT0000 complete [ 922.867435] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 925.998397] Key type lgssc unregistered [ 926.282104] LNet: 28770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.293141] LNetError: 28770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.304236] LNet: Removed LNI 192.168.204.145@tcp [ 927.305693] Key type .llcrypt unregistered [ 927.309113] Key type ._llcrypt unregistered [ 946.230397] Key type ._llcrypt registered [ 946.233539] Key type .llcrypt registered [ 946.343096] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 947.326363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 947.456414] alg: No test for adler32 (adler32-zlib) [ 948.506291] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 948.719426] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 950.392304] Key type lgssc registered [ 951.555489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.894483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.913276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.298530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.297796] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 971.746379] LustreError: 30040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.759638] LustreError: 30040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 974.671463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.094250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 980.426566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 991.843795] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 998.384269] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1005.126739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.629462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.642127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1013.362700] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1021.582655] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1040.353377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.362022] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.373037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.409914] Lustre: Skipped 1 previous similar message [ 1043.690709] Lustre: server umount lustre-OST0000 complete [ 1050.593795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.599733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.619805] Lustre: Skipped 1 previous similar message [ 1050.632581] Lustre: Skipped 2 previous similar messages [ 1055.717468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.742643] Lustre: Skipped 1 previous similar message [ 1060.834266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.846488] Lustre: Skipped 1 previous similar message [ 1062.367680] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.595260] Lustre: server umount lustre-MDT0000 complete [ 1065.959338] LustreError: 30041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.981185] LustreError: 30041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1066.857882] LustreError: 30021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186793 with bad export cookie 18403308927181303870 [ 1066.872988] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.882348] LustreError: 30021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1067.375077] Lustre: server umount lustre-MDT0001 complete [ 1078.439657] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1081.818399] Key type lgssc unregistered [ 1082.179390] LNet: 33779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.191375] LNetError: 33779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.213856] LNet: Removed LNI 192.168.204.145@tcp [ 1083.072730] Key type .llcrypt unregistered [ 1083.077824] Key type ._llcrypt unregistered [ 1102.392305] Key type ._llcrypt registered [ 1102.394220] Key type .llcrypt registered [ 1102.485852] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1103.472417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1103.530340] alg: No test for adler32 (adler32-zlib) [ 1104.719189] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1105.023131] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1106.743198] Key type lgssc registered [ 1107.867687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.674966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.701146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.158212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.822585] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1127.397050] LustreError: 35048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.424958] LustreError: 35048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1131.264544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.756624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1136.325363] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1144.289873] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1149.620546] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1155.933078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.287123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.429425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1162.783785] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1170.782051] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1183.201683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1183.207631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.224088] LustreError: Skipped 1 previous similar message [ 1183.226098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.269253] Lustre: Skipped 1 previous similar message [ 1186.604159] Lustre: server umount lustre-OST0000 complete [ 1193.440742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1193.441413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.467398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.476611] Lustre: Skipped 1 previous similar message [ 1193.492902] Lustre: Skipped 2 previous similar messages [ 1196.526489] Lustre: server umount lustre-MDT0000 complete [ 1198.561299] LustreError: 35043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.584227] LustreError: 35043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1200.209868] LustreError: 35029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784186927 with bad export cookie 3672107956234261460 [ 1200.221914] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1200.229025] LustreError: 35029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1200.582570] Lustre: server umount lustre-MDT0001 complete [ 1212.205732] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1214.854595] Key type lgssc unregistered [ 1215.185592] LNet: 38584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.198146] LNetError: 38584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.217099] LNet: Removed LNI 192.168.204.145@tcp [ 1215.967237] Key type .llcrypt unregistered [ 1215.970305] Key type ._llcrypt unregistered [ 1241.235218] Key type ._llcrypt registered [ 1241.239715] Key type .llcrypt registered [ 1241.386517] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1242.637081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.755970] alg: No test for adler32 (adler32-zlib) [ 1243.941804] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1244.254652] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1245.935784] Key type lgssc registered [ 1247.332975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1256.076454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1256.103436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.636526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.257732] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1267.169894] LustreError: 40001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.187773] LustreError: 40001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1271.721205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.223108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.290525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1285.675894] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1292.392923] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1300.230423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1300.612072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.644168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1305.882185] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1312.707865] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1323.489350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1323.503141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.520604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.789275] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1326.562966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.586299] Lustre: Skipped 1 previous similar message [ 1328.720480] Lustre: server umount lustre-MDT0000 complete [ 1330.883686] LustreError: 39996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.764167] LustreError: 39981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187059 with bad export cookie 3314843778447576326 [ 1332.766368] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.787755] LustreError: 39981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1336.022634] LustreError: 39995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.043357] LustreError: 39995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1336.058837] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 1336.066822] Lustre: Skipped 3 previous similar messages [ 1336.802912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.248807] Lustre: server umount lustre-MDT0001 complete [ 1354.468437] Lustre: server umount lustre-OST0000 complete [ 1370.524930] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1373.435461] Key type lgssc unregistered [ 1373.723540] LNet: 43862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.728540] LNetError: 43862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.759293] LNet: Removed LNI 192.168.204.145@tcp [ 1375.908140] Key type .llcrypt unregistered [ 1375.910135] Key type ._llcrypt unregistered [ 1400.386454] Key type ._llcrypt registered [ 1400.389202] Key type .llcrypt registered [ 1400.515843] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1401.558214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.649542] alg: No test for adler32 (adler32-zlib) [ 1402.643828] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1402.822345] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1404.481176] Key type lgssc registered [ 1405.695383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.981719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1414.001189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.412785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.999969] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1425.893763] LustreError: 45295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.918738] LustreError: 45295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1427.947722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.444230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.682753] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1439.706206] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1445.490882] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1452.014938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1452.286123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.359918] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1460.467716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1466.300692] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1473.239498] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1474.016794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1474.020278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.035670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.070570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1476.071814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.084562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.942612] Lustre: server umount lustre-OST0000 complete [ 1496.547012] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.550745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.561800] Lustre: Skipped 1 previous similar message [ 1496.573704] Lustre: Skipped 2 previous similar messages [ 1501.669307] LustreError: 47840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.683838] LustreError: 47840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1501.795576] Lustre: server umount lustre-MDT0000 complete [ 1502.689133] LustreError: 46024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.713359] LustreError: 46024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1505.627525] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187232 with bad export cookie 676931286381358531 [ 1505.634842] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.635323] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1512.065968] Lustre: server umount lustre-MDT0001 complete [ 1524.319781] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1528.900362] Key type lgssc unregistered [ 1529.260055] LNet: 49006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.265582] LNetError: 49006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.277870] LNet: Removed LNI 192.168.204.145@tcp [ 1530.151633] Key type .llcrypt unregistered [ 1530.166572] Key type ._llcrypt unregistered [ 1556.305358] Key type ._llcrypt registered [ 1556.308168] Key type .llcrypt registered [ 1556.469079] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1557.540958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1557.774232] alg: No test for adler32 (adler32-zlib) [ 1558.832285] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1559.074211] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1560.783211] Key type lgssc registered [ 1561.976237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1569.876723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1569.902679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.370188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.823217] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1581.540273] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.555518] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1583.756124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.076488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1588.779927] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1596.782392] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1603.586842] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1610.436904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.803153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.813491] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1616.956395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1624.792976] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1633.735749] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1637.292424] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1650.103531] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-16 3h36m13s [ 1651.939267] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1655.781427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1655.787988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.811229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.872174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.893219] Lustre: Skipped 4 previous similar messages [ 1658.626369] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1658.631791] Lustre: Skipped 1 previous similar message [ 1660.726548] Lustre: server umount lustre-MDT0000 complete [ 1662.949286] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.973617] LustreError: 50423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1663.874786] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1664.707323] LustreError: 50418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.098717] LustreError: 50418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.118681] LustreError: 50418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1670.314386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.459695] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.796353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.907360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1671.812615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1675.870359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1680.052569] Lustre: DEBUG MARKER: Wait for df (36711) ... [ 1682.052945] Lustre: DEBUG MARKER: done [ 1687.532311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.546372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.552232] Lustre: Skipped 3 previous similar messages [ 1687.561270] Lustre: Skipped 1 previous similar message [ 1692.642691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.650442] Lustre: Skipped 1 previous similar message [ 1692.858572] Lustre: server umount lustre-OST0000 complete [ 1697.767872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.786226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.963253] Lustre: server umount lustre-MDT0000 complete [ 1703.904886] LustreError: 50418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.934150] LustreError: 50418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1706.574227] LustreError: 50404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187433 with bad export cookie 13106207048102883509 [ 1706.580584] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.593294] LustreError: 50404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1706.929137] Lustre: server umount lustre-MDT0001 complete [ 1718.559419] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1721.954705] Key type lgssc unregistered [ 1722.312647] LNet: 55348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.316678] LNetError: 55348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.336474] LNet: Removed LNI 192.168.204.145@tcp [ 1723.390724] Key type .llcrypt unregistered [ 1723.392889] Key type ._llcrypt unregistered [ 1767.227152] Key type ._llcrypt registered [ 1767.229017] Key type .llcrypt registered [ 1767.319202] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1768.154300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1768.210948] alg: No test for adler32 (adler32-zlib) [ 1769.281565] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1769.476245] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1771.135774] Key type lgssc registered [ 1772.048357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.094694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.120638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.370813] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.392704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.510605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.762556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1793.390326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.559748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1793.617400] Lustre: 57560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1793.834849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1798.811009] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1806.070444] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.412638] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.864045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.058640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1818.272285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.848598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1824.374309] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1831.796837] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.734082] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-16 3h39m23s [ 1840.458987] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1848.086935] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1851.094257] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1851.199629] LustreError: 57565:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1871.595226] Lustre: lustre-MDT0000: Client 86f64d3c-dfc5-448e-acbe-d992adb2734e (at 192.168.204.45@tcp) reconnecting [ 1891.239117] LustreError: 57565:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1893.680037] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1896.176859] Lustre: DEBUG MARKER: done [ 1904.661903] Lustre: server umount lustre-OST0000 complete [ 1910.765137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.771841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.781895] Lustre: Skipped 1 previous similar message [ 1911.851634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.865633] Lustre: Skipped 2 previous similar messages [ 1917.925904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.932702] Lustre: Skipped 1 previous similar message [ 1919.022110] Lustre: server umount lustre-MDT0000 complete [ 1922.015116] Lustre: 56883:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187632/real 1784187632] req@00000000147a2ffa x1870856187862784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784187648 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1922.016962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.052859] Lustre: 56883:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1923.135776] LustreError: 57552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187650 with bad export cookie 10219611295588003010 [ 1923.144120] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.147061] LustreError: 57552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1923.481184] Lustre: server umount lustre-MDT0001 complete [ 1935.565933] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1938.753733] Key type lgssc unregistered [ 1939.085667] LNet: 61893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.099518] LNetError: 61893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.120731] LNet: Removed LNI 192.168.204.145@tcp [ 1940.099975] Key type .llcrypt unregistered [ 1940.103247] Key type ._llcrypt unregistered [ 1987.768417] Key type ._llcrypt registered [ 1987.770778] Key type .llcrypt registered [ 1987.873275] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1988.851685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1988.921680] alg: No test for adler32 (adler32-zlib) [ 1990.052424] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 1990.298238] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1991.983165] Key type lgssc registered [ 1992.947896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1996.388194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1996.411333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.713738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1997.769399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1998.029918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.047300] Lustre: Failing over lustre-MDT0000 [ 1999.403264] Lustre: server umount lustre-MDT0000 complete [ 2007.467881] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:42:12 (1784187732) [ 2015.247218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.823285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.403663] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2031.334899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.497201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2031.555673] Lustre: 64691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2031.908751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2036.524230] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2045.828085] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.401972] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.691809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.989343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2064.287398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.191204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2071.273153] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2079.737640] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2086.456927] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2093.536212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2093.547698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.581217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.097102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.109475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.135417] Lustre: Skipped 1 previous similar message [ 2099.846753] Lustre: server umount lustre-MDT0000 complete [ 2103.840049] LustreError: 64685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187830 with bad export cookie 10125461355256086429 [ 2103.854914] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.860578] LustreError: 64685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2104.235793] Lustre: server umount lustre-MDT0001 complete [ 2108.019976] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2113.047485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2122.655109] Lustre: 63616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187833/real 1784187833] req@000000007daf7f2c x1870856419584384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2122.672336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.686902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.767170] Lustre: 63615:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187835/real 1784187835] req@000000003ad38144 x1870856419584512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2127.903405] Lustre: 63615:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187838/real 1784187838] req@000000005631efb8 x1870856419584640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.874129] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8c84dd74a2fc0887 [ 2128.888991] Lustre: MGC192.168.204.145@tcp: Connection restored to 0@lo (at 0@lo) [ 2129.320949] LustreError: 66543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.346823] LustreError: 66543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2129.482294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.609799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2130.400394] Lustre: 63616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187840/real 1784187840] req@0000000052637a68 x1870856419584896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.489908] LustreError: 66544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.530897] LustreError: 66543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.564465] LustreError: 66543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2134.520345] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2134.540490] Lustre: 63617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187845/real 1784187845] req@00000000d6490d4f x1870856419585024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187861 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2137.568844] LustreError: 66544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.597883] LustreError: 66544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2142.705308] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.729200] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2143.950674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.249456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2148.495833] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2149.366925] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2151.468787] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2155.580911] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.969075] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2169.824329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2169.839644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.862475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.868437] Lustre: Skipped 2 previous similar messages [ 2171.877719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2171.892823] Lustre: Skipped 1 previous similar message [ 2174.780432] Lustre: server umount lustre-MDT0000 complete [ 2176.999340] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.010322] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2178.455483] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187905 with bad export cookie 10125461355256088711 [ 2178.462580] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.466863] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2178.911979] Lustre: server umount lustre-MDT0001 complete [ 2185.431466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2195.391259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.495203] Lustre: 63614:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784187909/real 1784187909] req@0000000049f157f4 x1870856419612800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784187925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.530910] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.548570] Lustre: Skipped 2 previous similar messages [ 2203.682954] LustreError: 63613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fb1794a7 x1870856419613952/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.143743] LustreError: 69200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.228549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.314957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2209.544973] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2218.215343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.444861] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2223.595502] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2227.692907] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2230.641625] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2237.291153] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2248.160837] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.176903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.185169] Lustre: Skipped 2 previous similar messages [ 2248.201019] Lustre: Skipped 2 previous similar messages [ 2251.237451] Lustre: server umount lustre-MDT0000 complete [ 2253.281596] LustreError: 71148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.315658] LustreError: 71148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2254.894311] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784187981 with bad export cookie 10125461355256090090 [ 2254.895793] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.903775] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2255.189200] Lustre: server umount lustre-MDT0001 complete [ 2261.336181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2265.172083] Lustre: DEBUG MARKER: files compared the same [ 2277.411513] Lustre: server umount lustre-OST0000 complete [ 2297.597786] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2301.691776] Key type lgssc unregistered [ 2302.103634] LNet: 74425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.110454] LNetError: 74425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.133819] LNet: Removed LNI 192.168.204.145@tcp [ 2303.190267] Key type .llcrypt unregistered [ 2303.192181] Key type ._llcrypt unregistered [ 2329.103668] Key type ._llcrypt registered [ 2329.105850] Key type .llcrypt registered [ 2329.226574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2330.183531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.198195] alg: No test for adler32 (adler32-zlib) [ 2331.253329] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2331.498349] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2333.183136] Key type lgssc registered [ 2334.287576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2342.650180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2342.692348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.208617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.552838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2354.660157] LustreError: 75841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.687236] LustreError: 75841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2358.022156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.431986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.304142] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2370.846682] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2377.243546] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.029373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.327570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.426228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2391.988510] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug +malloc all [ 2399.457540] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.052453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.056806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.084934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.176442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.181586] Lustre: Skipped 1 previous similar message [ 2419.510544] Lustre: server umount lustre-OST0000 complete [ 2424.292561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.304039] Lustre: Skipped 1 previous similar message [ 2424.309936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.313573] Lustre: Skipped 2 previous similar messages [ 2429.412178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.416167] Lustre: Skipped 1 previous similar message [ 2430.063520] Lustre: server umount lustre-MDT0000 complete [ 2434.458075] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188161 with bad export cookie 2371005705621002930 [ 2434.460077] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.465567] LustreError: 75823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2434.859911] Lustre: server umount lustre-MDT0001 complete [ 2448.135671] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2451.690682] Key type lgssc unregistered [ 2452.020133] LNet: 79394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.027650] LNetError: 79394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.048471] LNet: Removed LNI 192.168.204.145@tcp [ 2453.039730] Key type .llcrypt unregistered [ 2453.045586] Key type ._llcrypt unregistered [ 2480.444851] Key type ._llcrypt registered [ 2480.449652] Key type .llcrypt registered [ 2480.600642] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2481.696336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2481.840530] alg: No test for adler32 (adler32-zlib) [ 2483.223654] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2483.598570] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2485.367157] Key type lgssc registered [ 2486.661866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2494.794971] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2494.834940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.367537] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2515.487342] LustreError: 80787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 2542.175961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.520942] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2556.235586] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2564.561816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.728236] Lustre: *** cfs_fail_loc=706, val=0*** [ 2565.172691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.267717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2570.209724] LustreError: 80812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.222770] LustreError: 80812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2570.632440] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2575.338440] LustreError: 80813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.362263] LustreError: 80812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.387151] LustreError: 80812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2579.512955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.896576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2584.908988] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2594.565094] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2601.102950] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2605.536902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.565683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.568147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2605.583334] Lustre: Skipped 1 previous similar message [ 2609.534980] Lustre: server umount lustre-OST0000 complete [ 2615.780876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.801190] Lustre: Skipped 1 previous similar message [ 2615.816378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.830645] Lustre: Skipped 1 previous similar message [ 2620.301772] Lustre: server umount lustre-MDT0000 complete [ 2620.900249] LustreError: 81873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.923706] LustreError: 81873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2624.188456] LustreError: 80795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188351 with bad export cookie 1120579719552423174 [ 2624.201689] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.662244] Lustre: server umount lustre-MDT0001 complete [ 2637.324349] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2640.061719] Key type lgssc unregistered [ 2640.389827] LNet: 84403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2640.400641] LNetError: 84403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2640.459216] LNet: Removed LNI 192.168.204.145@tcp [ 2641.397150] Key type .llcrypt unregistered [ 2641.399714] Key type ._llcrypt unregistered [ 2668.192679] Key type ._llcrypt registered [ 2668.195727] Key type .llcrypt registered [ 2668.324448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2669.539293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2669.666034] alg: No test for adler32 (adler32-zlib) [ 2670.910638] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 2671.309816] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2673.063352] Key type lgssc registered [ 2674.544500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2683.925372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2683.952726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.871502] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2700.620037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.461985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.914280] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.933438] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2706.935195] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2712.033256] LustreError: 86509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.051874] LustreError: 86509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2716.338871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.972501] LustreError: 86510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.034376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.278371] LustreError: 86514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.303169] LustreError: 86514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2724.465548] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2727.408996] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.440810] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2734.192392] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.203378] Lustre: MGS: Client f60f5195-8e70-429b-925d-34a1630a8d53 (at 0@lo) reconnecting [ 2734.220952] LustreError: 86524:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2734.482690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.593075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2739.596202] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2755.557702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.577475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.604663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.162509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.167426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2760.192607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.521646] Lustre: server umount lustre-OST0000 complete [ 2770.401185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.419051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.427389] Lustre: Skipped 1 previous similar message [ 2770.434528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2770.444388] Lustre: Skipped 2 previous similar messages [ 2772.475292] Lustre: server umount lustre-MDT0000 complete [ 2775.520616] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.543945] LustreError: 86513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2776.815723] Lustre: server umount lustre-MDT0001 complete [ 2780.490749] Lustre: server umount MGS complete [ 2794.270819] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:55:18 (1784188518) [ 2829.355438] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 2841.118468] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2885.279387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.169200] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2902.059229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.288391] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2902.365758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2902.432204] Lustre: lustre-MDT0001: new disk, initializing [ 2902.533258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2902.559561] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2902.572297] Lustre: 93703:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2902.593829] LustreError: 93703:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2906.599027] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2916.692572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2916.880595] Lustre: lustre-OST0000: new disk, initializing [ 2916.884484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2916.890299] Lustre: 94468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2916.941530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.511827] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2927.712292] Lustre: 94927:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000078ba9987 x1870857133663488/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784188670 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2933.728513] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2942.854283] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.881021] Lustre: MGS: Client 5a7b8803-f8df-4359-8fc7-ab08500b57cc (at 0@lo) reconnecting [ 2942.953745] Lustre: 92969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2942.978135] LustreError: 93689:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2942.993708] LustreError: 93689:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2943.033631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2943.160808] Lustre: lustre-MDT0000: new disk, initializing [ 2943.410811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2943.449972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2943.473199] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2944.546855] Lustre: 95538:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004f0ca530 x1870857133670400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784188687 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2948.931513] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2949.736634] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2953.662640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2953.674940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2953.813684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2975.715455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2975.723960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.755848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2981.542099] Lustre: server umount lustre-OST0000 complete [ 2987.490216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2987.499609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2987.505933] Lustre: Skipped 2 previous similar messages [ 2987.527523] Lustre: Skipped 3 previous similar messages [ 2997.732407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2997.741986] Lustre: Skipped 3 previous similar messages [ 3000.287260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3000.445906] Lustre: server umount lustre-MDT0000 complete [ 3002.851233] LustreError: 93679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.879109] LustreError: 93679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3010.685858] Lustre: server umount lustre-MDT0001 complete [ 3014.870935] Lustre: server umount MGS complete [ 3028.639926] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:59:13 (1784188753) [ 3048.377461] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3051.289719] Key type lgssc unregistered [ 3051.663316] LNet: 98039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3051.676677] LNetError: 98039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3051.699226] LNet: Removed LNI 192.168.204.145@tcp [ 3052.632321] Key type .llcrypt unregistered [ 3052.638091] Key type ._llcrypt unregistered [ 3071.950875] Key type ._llcrypt registered [ 3071.954983] Key type .llcrypt registered [ 3072.095475] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3073.356671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.388237] alg: No test for adler32 (adler32-zlib) [ 3074.423824] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3074.773248] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3076.503205] Key type lgssc registered [ 3077.860985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3086.208177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3086.225440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.780834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.122717] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3097.067310] LustreError: 99260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.096512] LustreError: 99260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3102.182282] LustreError: 99259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.994114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.468817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3109.785211] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3119.518477] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3126.519640] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.268727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.749970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.280078] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3154.143751] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3157.308281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3169.762520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3169.779557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3169.799311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3172.832740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3172.837339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3172.850871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.900779] Lustre: Skipped 1 previous similar message [ 3175.058755] Lustre: server umount lustre-OST0000 complete [ 3181.536900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3181.545938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.567607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.697730] Lustre: server umount lustre-MDT0000 complete [ 3188.193884] LustreError: 99260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.234686] LustreError: 99260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3189.904858] LustreError: 99240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784188916 with bad export cookie 779595793631235791 [ 3189.917684] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.337347] Lustre: server umount lustre-MDT0001 complete [ 3202.377721] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3205.052836] Key type lgssc unregistered [ 3205.355540] LNet: 102830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3205.365473] LNetError: 102830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3205.377555] LNet: Removed LNI 192.168.204.145@tcp [ 3206.340477] Key type .llcrypt unregistered [ 3206.344164] Key type ._llcrypt unregistered [ 3223.607066] Key type ._llcrypt registered [ 3223.610269] Key type .llcrypt registered [ 3223.748684] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3225.143243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3225.255152] alg: No test for adler32 (adler32-zlib) [ 3226.395612] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3226.748536] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3228.431160] Key type lgssc registered [ 3229.925477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3237.128077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3237.136205] LustreError: 103913:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3237.156663] LustreError: 103917:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3237.161704] LustreError: 103913:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3237.173170] LustreError: 103913:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3237.179573] LustreError: 103913:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3237.184075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.691570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.955243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.097086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3248.207144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3253.336749] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3259.291111] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing lsmod [ 3262.499417] LustreError: 105119:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3262.531824] LustreError: 105123:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3262.544505] LustreError: 105119:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4968 [ 3262.548970] LustreError: 105119:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3262.558097] LustreError: 105119:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3262.575413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.086773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.139980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3268.577192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3268.599745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.636976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.611879] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.625183] Lustre: Skipped 1 previous similar message [ 3273.766448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.775735] Lustre: Skipped 5 previous similar messages [ 3278.818735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.836320] Lustre: Skipped 2 previous similar messages [ 3280.351179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3280.607798] Lustre: server umount lustre-MDT0000 complete [ 3283.941357] LustreError: 103950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.963482] LustreError: 103950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3284.565055] LustreError: 103929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189011 with bad export cookie 7837391379885496492 [ 3284.574443] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3284.579077] LustreError: 103929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3286.111403] LustreError: 104385:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007d0bc875 x1870857715603968/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3286.291257] Lustre: server umount lustre-MDT0001 complete [ 3300.812161] Lustre: server umount lustre-OST0000 complete [ 3309.761645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.410150] LustreError: 106191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.531098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.679930] LustreError: 106192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.993883] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3320.804428] LustreError: 106191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.666160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.041688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3329.619506] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3336.485615] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3342.821071] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3349.797663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.928160] LustreError: 108011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.141575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.254665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3356.761482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3365.933317] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3375.763297] Lustre: server umount lustre-OST0000 complete [ 3381.225754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.255292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3381.257735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.281424] Lustre: Skipped 1 previous similar message [ 3389.546377] Lustre: server umount lustre-MDT0000 complete [ 3391.469659] LustreError: 106192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3391.501240] LustreError: 106192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3392.352432] Lustre: 103317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784189103/real 1784189103] req@00000000297910d2 x1870857715631616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784189119 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3392.404227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3392.420231] Lustre: Skipped 1 previous similar message [ 3394.119963] LustreError: 106171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189121 with bad export cookie 7837391379885497416 [ 3394.123082] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3394.138473] LustreError: 106171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3394.599543] Lustre: server umount lustre-MDT0001 complete [ 3406.637823] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3409.483978] Key type lgssc unregistered [ 3409.762542] LNet: 109725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.774463] LNetError: 109725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3409.791381] LNet: Removed LNI 192.168.204.145@tcp [ 3410.756383] Key type .llcrypt unregistered [ 3410.759137] Key type ._llcrypt unregistered [ 3435.193973] Key type ._llcrypt registered [ 3435.198557] Key type .llcrypt registered [ 3435.358855] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3437.034984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3437.133024] alg: No test for adler32 (adler32-zlib) [ 3438.399770] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3438.662413] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3440.393261] Key type lgssc registered [ 3441.652571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3449.761330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3449.812394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.330130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.882312] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3461.609348] LustreError: 111140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.663708] LustreError: 111140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3465.791363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.386237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.020901] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3481.199578] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3488.623543] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3495.918153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.281377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.328703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3501.916807] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3509.257196] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3517.937947] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3523.359598] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3532.270867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.280724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.285365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3532.301180] Lustre: Skipped 1 previous similar message [ 3534.909724] Lustre: server umount lustre-OST0000 complete [ 3542.502889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.506262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3542.535266] Lustre: Skipped 2 previous similar messages [ 3545.692671] Lustre: server umount lustre-MDT0000 complete [ 3547.621408] LustreError: 111141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.640524] LustreError: 111141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3549.816858] LustreError: 111120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189276 with bad export cookie 71305195798027202 [ 3549.821182] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.832305] LustreError: 111120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3550.249416] Lustre: server umount lustre-MDT0001 complete [ 3563.469449] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3566.639658] Key type lgssc unregistered [ 3566.945631] LNet: 114783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3566.964734] LNetError: 114783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3566.992891] LNet: Removed LNI 192.168.204.145@tcp [ 3567.921942] Key type .llcrypt unregistered [ 3567.924425] Key type ._llcrypt unregistered [ 3587.139684] Key type ._llcrypt registered [ 3587.141759] Key type .llcrypt registered [ 3587.287077] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3588.519235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3588.719384] alg: No test for adler32 (adler32-zlib) [ 3589.789468] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3590.007977] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3591.703842] Key type lgssc registered [ 3592.852233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3601.684704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3601.727359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.177026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.683073] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3613.677389] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.695188] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3618.615233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.795252] LustreError: 116007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3619.149375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3624.221402] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3633.483099] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3640.530641] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3647.920103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3648.334192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.401900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3656.447272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3665.916826] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3680.241213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3680.245098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.254141] LustreError: Skipped 1 previous similar message [ 3680.259802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3680.279600] Lustre: Skipped 1 previous similar message [ 3680.297379] Lustre: Skipped 1 previous similar message [ 3682.071667] Lustre: server umount lustre-OST0000 complete [ 3690.464592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.469136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.472959] Lustre: Skipped 1 previous similar message [ 3690.480438] Lustre: Skipped 1 previous similar message [ 3692.956288] Lustre: server umount lustre-MDT0000 complete [ 3695.596046] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.630497] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3697.755056] LustreError: 115986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189424 with bad export cookie 12752503843675889572 [ 3697.761814] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3698.114846] Lustre: server umount lustre-MDT0001 complete [ 3711.089204] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3714.176634] Key type lgssc unregistered [ 3714.531492] LNet: 119540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3714.551447] LNetError: 119540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3714.579596] LNet: Removed LNI 192.168.204.145@tcp [ 3715.527456] Key type .llcrypt unregistered [ 3715.529755] Key type ._llcrypt unregistered [ 3743.228739] Key type ._llcrypt registered [ 3743.231111] Key type .llcrypt registered [ 3743.367659] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3744.589317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3744.603186] alg: No test for adler32 (adler32-zlib) [ 3745.801869] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3746.222241] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3747.928197] Key type lgssc registered [ 3749.226927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3757.719065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3757.755166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.264224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.474784] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3769.837510] LustreError: 120957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.857276] LustreError: 120957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3773.618414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.043470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3780.040494] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3789.811937] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3797.751258] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3806.095396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3806.650545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.139165] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3822.957941] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3829.183084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3840.841974] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3845.127086] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3845.133354] Lustre: Skipped 2 previous similar messages [ 3847.310555] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3847.357889] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3850.557928] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3850.564577] Lustre: Skipped 2 previous similar messages [ 3857.025052] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3857.037526] Lustre: Skipped 1 previous similar message [ 3863.288342] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3863.296087] Lustre: Skipped 2 previous similar messages [ 3871.885285] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.45@tcp 192.168.0.[2,10]@tcp [ 3876.704893] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.45/24@tcp in log lustre-MDT0000 [ 3876.717480] Lustre: Skipped 2 previous similar messages [ 3878.225246] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 3878.236803] Lustre: Skipped 1 previous similar message [ 3890.657601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3890.660965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.664940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3890.687179] Lustre: Skipped 1 previous similar message [ 3890.718085] Lustre: Skipped 1 previous similar message [ 3893.583497] Lustre: server umount lustre-OST0000 complete [ 3900.914077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.935950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.963762] Lustre: Skipped 1 previous similar message [ 3905.736445] Lustre: server umount lustre-MDT0000 complete [ 3906.048083] LustreError: 120957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.093951] LustreError: 120957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3910.181035] LustreError: 123824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.208734] LustreError: 123824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3910.836989] LustreError: 120939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189637 with bad export cookie 4535471519998388052 [ 3910.846953] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.858567] LustreError: 120939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3911.568303] Lustre: server umount lustre-MDT0001 complete [ 3926.917197] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3931.002600] Key type lgssc unregistered [ 3931.375728] LNet: 125928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3931.383055] LNetError: 125928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3932.453088] LNet: Removed LNI 192.168.204.145@tcp [ 3933.690807] Key type .llcrypt unregistered [ 3933.695950] Key type ._llcrypt unregistered [ 3968.146814] Key type ._llcrypt registered [ 3968.148540] Key type .llcrypt registered [ 3968.322546] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3970.279377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.306156] alg: No test for adler32 (adler32-zlib) [ 3971.593993] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 3972.027816] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3973.799280] Key type lgssc registered [ 3975.003883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.090976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.122344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.892512] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3985.935887] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.45/24@tcp [ 3985.952323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.621990] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3995.108118] LustreError: 127496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.147495] LustreError: 127496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4000.225947] LustreError: 127497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4002.970812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.821792] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4003.899930] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.45/24@tcp [ 4003.985282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4010.404624] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4020.419204] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4028.247507] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4036.947872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4037.470991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.363518] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4054.543205] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4055.922806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4071.395045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4071.413608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.420691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4071.434516] Lustre: Skipped 1 previous similar message [ 4071.449989] Lustre: Skipped 1 previous similar message [ 4075.855462] Lustre: server umount lustre-OST0000 complete [ 4080.611350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4080.625510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.647128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.634161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.660890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.664911] Lustre: Skipped 1 previous similar message [ 4086.759512] LustreError: 127492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.771776] LustreError: 127492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4086.800482] Lustre: server umount lustre-MDT0000 complete [ 4091.028047] LustreError: 127477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784189818 with bad export cookie 8866290771317268795 [ 4091.033619] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.038805] LustreError: 127477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4091.461320] Lustre: server umount lustre-MDT0001 complete [ 4106.300078] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4110.459521] Key type lgssc unregistered [ 4110.860037] LNet: 131093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4110.875166] LNetError: 131093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4111.917269] LNet: Removed LNI 192.168.204.145@tcp [ 4112.615325] Key type .llcrypt unregistered [ 4112.617323] Key type ._llcrypt unregistered [ 4148.296795] Key type ._llcrypt registered [ 4148.305759] Key type .llcrypt registered [ 4148.462727] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4170.707323] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4172.777296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4172.896437] alg: No test for adler32 (adler32-zlib) [ 4174.095878] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 4174.635773] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4176.440355] Key type lgssc registered [ 4179.186317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4225.846162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4225.888304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.156535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4227.188118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4227.285564] Lustre: lustre-MDT0000: new disk, initializing [ 4227.410800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4227.459191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4232.204614] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4244.177655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.339910] Lustre: 134497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4244.405822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4244.415244] Lustre: Skipped 1 previous similar message [ 4244.521684] Lustre: lustre-MDT0001: new disk, initializing [ 4244.642374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.691152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4244.705304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4250.141473] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4262.197240] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4269.523938] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4278.328652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4278.731883] Lustre: lustre-OST0000: new disk, initializing [ 4278.744232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4278.758770] Lustre: 136467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4278.920197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.583643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4285.600303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4285.780175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4287.051190] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4299.004771] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4305.262679] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4305.507289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4311.658574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4311.905398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4316.640197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4316.653188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.661614] LustreError: Skipped 1 previous similar message [ 4316.664777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4316.683614] Lustre: Skipped 1 previous similar message [ 4320.582857] Lustre: server umount lustre-OST0000 complete [ 4326.881738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.883706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4326.908469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.916771] Lustre: Skipped 2 previous similar messages [ 4331.510491] Lustre: server umount lustre-MDT0000 complete [ 4335.976193] LustreError: 134491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190062 with bad export cookie 12401560308713441916 [ 4335.986041] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4336.006953] LustreError: 134491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4336.533743] Lustre: server umount lustre-MDT0001 complete [ 4347.820164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.353629] LustreError: 138689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.396867] LustreError: 138689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4348.515323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.505132] LustreError: 138690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.432073] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4358.630670] LustreError: 138689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.746534] LustreError: 138690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.955773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.357636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4371.081574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4379.314210] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4387.074852] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4394.952733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4395.504723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.540654] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4411.924987] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4417.964582] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4419.362337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4425.171990] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4425.476730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4435.707159] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4435.890727] Lustre: lustre-OST0001: new disk, initializing [ 4435.894626] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4435.901479] Lustre: 142069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4435.992299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4443.626255] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4445.895478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4445.914123] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4446.024768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4453.782213] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4454.014327] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4459.340608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4459.668218] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4466.641129] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff958f06b6c800.ost_server_uuid 50 [ 4468.660169] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff958f06b6c800.ost_server_uuid in IDLE state after 0 sec [ 4482.020690] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4482.028048] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.034319] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4482.049864] Lustre: Skipped 2 previous similar messages [ 4482.075627] Lustre: Skipped 1 previous similar message [ 4484.684237] Lustre: server umount lustre-OST0001 complete [ 4487.140379] LustreError: 140511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.158398] LustreError: 140511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4489.697857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4489.713526] LustreError: Skipped 1 previous similar message [ 4489.720694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.739104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4495.376533] Lustre: server umount lustre-OST0000 complete [ 4502.500144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.512519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.516257] Lustre: Skipped 2 previous similar messages [ 4502.530241] Lustre: Skipped 3 previous similar messages [ 4506.316592] Lustre: server umount lustre-MDT0000 complete [ 4507.620451] LustreError: 138686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.636020] LustreError: 138686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4510.448298] LustreError: 138670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190237 with bad export cookie 12401560308713442798 [ 4510.452373] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4510.464443] LustreError: 138670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4510.858238] Lustre: server umount lustre-MDT0001 complete [ 4532.238483] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4535.425737] Key type lgssc unregistered [ 4535.778540] LNet: 145071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4535.799818] LNetError: 145071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4535.847813] LNet: Removed LNI 192.168.204.145@tcp [ 4536.862161] Key type .llcrypt unregistered [ 4536.866223] Key type ._llcrypt unregistered [ 4584.943670] Key type ._llcrypt registered [ 4584.945901] Key type .llcrypt registered [ 4585.055927] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4600.890400] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4602.334345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4602.357169] alg: No test for adler32 (adler32-zlib) [ 4603.414588] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 4603.770963] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4605.487229] Key type lgssc registered [ 4606.876735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4647.786826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4647.820288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4649.178808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4649.229505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4649.324501] Lustre: lustre-MDT0000: new disk, initializing [ 4649.446664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.470981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4654.717398] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4667.555959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.730528] Lustre: 149230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4667.801874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4667.805361] Lustre: Skipped 1 previous similar message [ 4667.878224] Lustre: lustre-MDT0001: new disk, initializing [ 4668.006483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4668.075106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4668.092364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4673.806565] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4683.944513] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4689.620591] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4697.310958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.630177] Lustre: lustre-OST0000: new disk, initializing [ 4697.639623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4697.658424] Lustre: 151196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4697.757132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4704.384634] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4705.891833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4705.905384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4706.049839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4715.542139] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4720.337654] Lustre: Failing over lustre-OST0000 [ 4720.666121] Lustre: server umount lustre-OST0000 complete [ 4738.016332] Lustre: 147453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190448/real 1784190448] req@00000000164d1c5a x1870859159505152/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784190464 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4738.022386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.041652] Lustre: 147453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4741.877170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4742.175136] Lustre: 147454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190453/real 1784190453] req@0000000092180b40 x1870859159506432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784190469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4742.216212] Lustre: 147454:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4742.469569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4742.491673] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4744.479165] Lustre: 147455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190453/real 1784190453] req@000000005030c535 x1870859159506560/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784190469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4747.235392] Lustre: 147454:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190458/real 1784190458] req@0000000046e95042 x1870859159507584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784190474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4747.976928] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4748.635382] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4748.639298] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4750.135291] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4752.737525] Lustre: 147453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190463/real 1784190463] req@000000001fb6adb1 x1870859159508736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784190479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4752.769642] Lustre: 147453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4754.863626] Lustre: Failing over lustre-MDT0000 [ 4754.912248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4754.922657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.946064] Lustre: Skipped 1 previous similar message [ 4754.953315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4755.178296] Lustre: server umount lustre-MDT0000 complete [ 4759.010938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.041489] Lustre: Skipped 1 previous similar message [ 4763.368246] LustreError: 149237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.412057] LustreError: 149237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4764.133650] LustreError: 149238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4764.156600] LustreError: 149238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4768.463056] LustreError: 150263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.569608] LustreError: 149236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4773.595024] LustreError: 149236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4774.369428] Lustre: 147453:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784190485/real 1784190485] req@00000000fd25ec0d x1870859159520256/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784190501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4774.399029] Lustre: 147453:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4774.406965] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.314797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.561147] LustreError: 150263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.600887] LustreError: 150263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4784.616871] LustreError: 147451:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000033583779 x1870859159525504/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4784.963733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.041247] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4788.941065] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4790.084184] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4790.268482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4790.280904] Lustre: Skipped 1 previous similar message [ 4790.308100] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4790.381838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4799.980378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4800.003839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.023994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4805.601818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4805.612518] Lustre: Skipped 3 previous similar messages [ 4806.147254] Lustre: server umount lustre-OST0000 complete [ 4810.720923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.733223] Lustre: Skipped 1 previous similar message [ 4810.739979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4810.746562] Lustre: Skipped 1 previous similar message [ 4815.849653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.862433] Lustre: Skipped 1 previous similar message [ 4818.606637] Lustre: server umount lustre-MDT0000 complete [ 4820.960204] LustreError: 150263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.984523] LustreError: 150263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4823.532222] LustreError: 151207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784190550 with bad export cookie 2367157782188591867 [ 4823.547156] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4823.556067] LustreError: 151207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4823.952243] Lustre: server umount lustre-MDT0001 complete [ 4838.401927] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4841.745254] Key type lgssc unregistered [ 4842.187477] LNet: 154856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4842.210279] LNetError: 154856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4842.253680] LNet: Removed LNI 192.168.204.145@tcp [ 4843.256207] Key type .llcrypt unregistered [ 4843.257848] Key type ._llcrypt unregistered [ 4881.082430] Key type ._llcrypt registered [ 4881.086781] Key type .llcrypt registered [ 4881.200566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4882.181774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4882.365305] alg: No test for adler32 (adler32-zlib) [ 4883.422298] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 4883.675344] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4885.343194] Key type lgssc registered [ 4886.426219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4893.688065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4893.714236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.336731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4899.830849] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4905.955442] LustreError: 156387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.971736] LustreError: 156387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4909.620696] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4911.082062] LustreError: 156386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.841505] LustreError: 156416:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4913.861840] LustreError: 156416:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4913.878415] LustreError: 156416:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4914.294762] Lustre: server umount lustre-MDT0000 complete [ 4930.264966] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4941.307443] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4981.453911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4981.713385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4981.738406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4981.824359] Lustre: lustre-MDT0000: new disk, initializing [ 4981.914421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.945725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4986.466282] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4998.116636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.228845] Lustre: 159723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4998.370778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4998.373896] Lustre: Skipped 1 previous similar message [ 4998.469784] Lustre: lustre-MDT0001: new disk, initializing [ 4998.586235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4998.640287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4998.656527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5003.312621] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5013.278199] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5019.120476] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5026.402718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.695348] Lustre: lustre-OST0000: new disk, initializing [ 5026.706042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5026.717246] Lustre: 161692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5026.839700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5034.086697] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5037.124376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5037.134247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5037.275948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5044.938281] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5050.068936] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5052.196633] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:32:56 (1784190776) [ 5349.855717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.856843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5349.866490] Lustre: Skipped 2 previous similar messages [ 5349.876251] Lustre: Skipped 2 previous similar messages [ 5353.376354] Lustre: server umount lustre-MDT0000 complete [ 5354.979425] LustreError: 160590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5354.998394] LustreError: 160590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5357.311901] LustreError: 159713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191084 with bad export cookie 9425335452215745019 [ 5357.325854] LustreError: 159713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5357.327156] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5360.096115] LustreError: 159728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.097855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.139497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5362.144483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5363.805947] Lustre: server umount lustre-MDT0001 complete [ 5374.768322] Lustre: server umount lustre-OST0000 complete [ 5385.619829] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5396.463546] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5437.994932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.219339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5438.247423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5438.341803] Lustre: lustre-MDT0000: new disk, initializing [ 5438.414590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.428460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5443.392291] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5453.983942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5454.094298] Lustre: 166363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5454.178703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5454.190043] Lustre: Skipped 1 previous similar message [ 5454.293407] Lustre: lustre-MDT0001: new disk, initializing [ 5454.403493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.440482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5454.454800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5459.464259] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5470.067221] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5477.441199] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5487.626305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5488.067144] Lustre: lustre-OST0000: new disk, initializing [ 5488.071182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5488.077464] Lustre: 168331:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5488.172842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.419670] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5497.908822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5497.915861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5498.057472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5505.284053] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5510.229644] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5510.526978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5515.467535] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5515.663237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5518.818752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5518.833942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.836618] LustreError: Skipped 1 previous similar message [ 5518.855328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5518.862692] Lustre: Skipped 1 previous similar message [ 5523.798962] Lustre: server umount lustre-OST0000 complete [ 5529.060425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.069924] Lustre: Skipped 1 previous similar message [ 5529.075290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5529.083874] Lustre: Skipped 1 previous similar message [ 5538.275460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5538.291861] Lustre: Skipped 3 previous similar messages [ 5542.367156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5542.738335] Lustre: server umount lustre-MDT0000 complete [ 5543.405282] LustreError: 166374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.430733] LustreError: 166374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5547.695390] LustreError: 166355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191274 with bad export cookie 9425335452215903849 [ 5547.700127] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5547.706850] LustreError: 166355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5548.301847] Lustre: server umount lustre-MDT0001 complete [ 5560.641397] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:41:25 (1784191285) [ 5576.137872] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5586.170121] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5629.890360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.150989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5630.189274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5630.278539] Lustre: lustre-MDT0000: new disk, initializing [ 5630.383778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.402658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5636.179957] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5647.807542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.053274] Lustre: 173091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5648.519303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5653.878122] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5664.685700] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5672.182548] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5680.156789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5680.582541] Lustre: lustre-OST0000: new disk, initializing [ 5680.588168] Lustre: Skipped 1 previous similar message [ 5680.604173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5680.619338] Lustre: Skipped 2 previous similar messages [ 5680.625228] Lustre: 175070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5687.498113] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5688.891420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5688.906934] Lustre: Skipped 1 previous similar message [ 5688.910792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5689.026549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5697.633755] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5709.794637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.806955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5709.835864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5711.893982] Lustre: server umount lustre-OST0000 complete [ 5720.041767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5720.045912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.074875] Lustre: Skipped 2 previous similar messages [ 5722.408427] Lustre: server umount lustre-MDT0000 complete [ 5725.153473] LustreError: 173098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5725.176554] LustreError: 173098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5726.584684] LustreError: 173083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191453 with bad export cookie 9425335452215904731 [ 5726.590197] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5726.593965] LustreError: 173083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5727.064605] Lustre: server umount lustre-MDT0001 complete [ 5737.111802] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:44:21 (1784191461) [ 5755.681557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 5766.314575] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5805.800562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.087684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5806.135839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5806.265578] Lustre: lustre-MDT0000: new disk, initializing [ 5806.348455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5806.351710] Lustre: Skipped 2 previous similar messages [ 5806.392091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5811.542556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5822.474233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.584965] Lustre: 179396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5822.838913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5827.731765] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5837.832919] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5843.710206] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5851.715441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5852.126530] Lustre: 181374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5858.402205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5858.559373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5860.172535] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5871.881050] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5884.390521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5884.391108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.395202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5884.395210] Lustre: Skipped 3 previous similar messages [ 5884.398597] LustreError: Skipped 1 previous similar message [ 5884.425728] Lustre: Skipped 1 previous similar message [ 5887.624210] Lustre: server umount lustre-OST0000 complete [ 5894.627590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5906.403352] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5906.611968] Lustre: server umount lustre-MDT0000 complete [ 5909.991701] LustreError: 179403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5910.004428] LustreError: 179403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5910.859891] LustreError: 181373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191637 with bad export cookie 9425335452215905809 [ 5910.868092] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.883869] LustreError: 181373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5911.376889] Lustre: server umount lustre-MDT0001 complete [ 5923.993326] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5927.712561] Key type lgssc unregistered [ 5928.181378] LNet: 183232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5928.195274] LNetError: 183232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5928.209211] LNet: Removed LNI 192.168.204.145@tcp [ 5929.495226] Key type .llcrypt unregistered [ 5929.497519] Key type ._llcrypt unregistered [ 5956.647199] Key type ._llcrypt registered [ 5956.651159] Key type .llcrypt registered [ 5956.809556] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5958.120812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5958.247283] alg: No test for adler32 (adler32-zlib) [ 5959.357631] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 5959.680322] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5961.439360] Key type lgssc registered [ 5962.825184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5972.811863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5972.854482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.387396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5979.406766] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5984.751597] LustreError: 184652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.765408] LustreError: 184652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5988.488371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.024082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5994.135380] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6001.563754] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6006.465591] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6015.057434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.542118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.646638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6021.858764] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6031.462150] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6054.371161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.372600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.401614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.453165] Lustre: Skipped 1 previous similar message [ 6058.746790] Lustre: server umount lustre-OST0000 complete [ 6064.624866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.631595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.661177] Lustre: Skipped 1 previous similar message [ 6064.672116] Lustre: Skipped 1 previous similar message [ 6069.713137] Lustre: server umount lustre-MDT0000 complete [ 6069.794541] LustreError: 187156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.813068] LustreError: 187156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6074.064495] LustreError: 185389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191801 with bad export cookie 1411660763371466442 [ 6074.081747] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.086293] LustreError: 185389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6074.586676] Lustre: server umount lustre-MDT0001 complete [ 6085.257132] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6088.338559] Key type lgssc unregistered [ 6088.608668] LNet: 188213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6088.620482] LNetError: 188213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6088.650192] LNet: Removed LNI 192.168.204.145@tcp [ 6089.508213] Key type .llcrypt unregistered [ 6089.512855] Key type ._llcrypt unregistered [ 6116.744390] Key type ._llcrypt registered [ 6116.746665] Key type .llcrypt registered [ 6116.945362] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6117.853751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.865502] alg: No test for adler32 (adler32-zlib) [ 6119.027973] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 6119.396710] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6121.103180] Key type lgssc registered [ 6122.642420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6132.445788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6132.463109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.973631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.871830] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6143.464849] LustreError: 189647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.495364] LustreError: 189647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6148.569366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.578775] LustreError: 189648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6149.193532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6154.602955] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6164.722543] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6172.744403] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6181.129515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.509830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6188.654873] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6197.918061] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6203.081416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6210.534331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6210.541383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.561893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6213.602991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6213.613405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6213.620672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.378992] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 6215.394882] Lustre: Skipped 1 previous similar message [ 6216.857753] Lustre: server umount lustre-OST0000 complete [ 6224.405696] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff958f0964c000.ost_server_uuid 50 [ 6226.375614] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6243.298238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.319047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.330623] Lustre: Skipped 1 previous similar message [ 6248.097547] Lustre: server umount lustre-MDT0000 complete [ 6248.416679] LustreError: 189642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.434991] LustreError: 189642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6252.190837] LustreError: 189627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784191979 with bad export cookie 5219221004275925867 [ 6252.206244] LustreError: 189627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6252.206597] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6252.649289] Lustre: server umount lustre-MDT0001 complete [ 6263.314857] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:53:08 (1784191988) [ 6270.922085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6271.371083] LustreError: 193585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.554552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.594767] LustreError: 193586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.448129] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6287.854650] LustreError: 193586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.885704] LustreError: 193586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6290.827672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.706248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6296.944767] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6307.026313] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6313.459675] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6321.488358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6322.025097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.524327] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6338.208472] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6343.534611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6345.642485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6345.898546] Lustre: lustre-OST0001: new disk, initializing [ 6345.901856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6345.910111] Lustre: 196428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6345.990165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6353.045403] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6354.068391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6354.095976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6354.252916] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6363.776815] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6367.200251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6367.212730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.225754] Lustre: Skipped 1 previous similar message [ 6367.229722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.259931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6372.883257] Lustre: server umount lustre-OST0000 complete [ 6374.608450] LustreError: 195410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.090628] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6381.362770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6387.447052] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6387.668749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6405.602269] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6405.611285] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6405.620440] LustreError: Skipped 1 previous similar message [ 6405.624067] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6405.624079] Lustre: Skipped 3 previous similar messages [ 6405.700513] Lustre: Skipped 2 previous similar messages [ 6408.295299] Lustre: server umount lustre-OST0001 complete [ 6414.305398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6414.334808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.850853] Lustre: server umount lustre-MDT0000 complete [ 6420.960963] LustreError: 193585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.975439] LustreError: 193585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6423.188855] LustreError: 196427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192150 with bad export cookie 5219221004275926938 [ 6423.197519] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.202690] LustreError: 196427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6423.693889] Lustre: server umount lustre-MDT0001 complete [ 6452.378774] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:56:16 (1784192176) [ 6459.851214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.298586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6460.353984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6460.650345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.287274] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6475.174895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.257587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6475.278147] Lustre: 199869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6483.441388] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6496.231602] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6503.387682] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6511.411622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6511.667241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6517.474743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6519.020549] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6528.040313] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.618607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.742811] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6535.973433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6535.992768] Lustre: Skipped 2 previous similar messages [ 6543.008954] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6543.295979] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6551.652568] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6557.418680] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 6557.425504] Lustre: Skipped 4 previous similar messages [ 6558.708759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.734799] Lustre: Skipped 2 previous similar messages [ 6558.741821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6560.547748] Lustre: server umount lustre-OST0000 complete [ 6562.501700] LustreError: 201722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.557589] Lustre: server umount lustre-OST0001 complete [ 6594.526801] Lustre: server umount lustre-MDT0000 complete [ 6598.965878] LustreError: 200616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192325 with bad export cookie 5219221004275928345 [ 6598.986787] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6598.998667] LustreError: 200616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6599.349944] Lustre: server umount lustre-MDT0001 complete [ 6626.541890] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:59:11 (1784192351) [ 6644.239740] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 6656.095156] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6695.552712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.807756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6695.839873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6695.921898] Lustre: lustre-MDT0000: new disk, initializing [ 6696.026538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.042158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6700.429196] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6710.957151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.110573] Lustre: 207967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6711.186899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6711.189879] Lustre: Skipped 1 previous similar message [ 6711.235652] Lustre: lustre-MDT0001: new disk, initializing [ 6711.393874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6711.409867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6716.209274] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6727.436263] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6735.173239] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6743.756510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6744.187606] Lustre: lustre-OST0000: new disk, initializing [ 6744.197348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6744.209234] Lustre: 209950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6751.310281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6751.321154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6751.437827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6752.041442] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6761.370710] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6768.560463] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6768.776285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6774.026540] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6774.276541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6777.314897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6777.316624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.333850] LustreError: Skipped 4 previous similar messages [ 6777.353405] Lustre: Skipped 4 previous similar messages [ 6777.355645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6777.390911] Lustre: Skipped 9 previous similar messages [ 6782.803680] Lustre: server umount lustre-OST0000 complete [ 6793.593508] Lustre: server umount lustre-MDT0000 complete [ 6797.796862] LustreError: 207979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.830523] LustreError: 207979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6799.269156] LustreError: 207961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192526 with bad export cookie 5219221004275930130 [ 6799.285528] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.294082] LustreError: 207961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6799.962638] Lustre: server umount lustre-MDT0001 complete [ 6810.195627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.305570] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6826.352371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.052440] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6841.026085] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6847.771272] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6855.774457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.870901] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6872.857307] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6878.261212] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6881.907583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6886.757432] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6886.938889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6891.488207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6891.496290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.513864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6891.528750] Lustre: Skipped 4 previous similar messages [ 6891.538289] Lustre: Skipped 6 previous similar messages [ 6903.263172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6903.488967] Lustre: server umount lustre-OST0000 complete [ 6910.538366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6915.690838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6945.468111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.064127] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6956.067516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6956.080564] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6956.107211] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6956.119721] Lustre: Skipped 1 previous similar message [ 6963.344907] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.981163] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6978.501656] Lustre: server umount lustre-OST0000 complete [ 6989.522838] Lustre: server umount lustre-MDT0000 complete [ 6994.319114] LustreError: 212148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192721 with bad export cookie 5219221004275931012 [ 6994.355467] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6994.804393] Lustre: server umount lustre-MDT0001 complete [ 7004.501444] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:05:29 (1784192729) [ 7011.327225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.874550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.886474] Lustre: Skipped 6 previous similar messages [ 7017.272893] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7026.001605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.742665] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7039.439209] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7045.788162] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7053.579057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7061.414220] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7070.947421] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7075.986634] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7076.207926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7081.364941] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7081.638515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7089.490619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7089.712539] Lustre: lustre-OST0001: new disk, initializing [ 7089.716591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7089.724562] Lustre: 221530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7096.941507] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7097.482269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7097.497620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7097.604541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7108.432869] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.457838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7113.755535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7119.978071] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7120.229758] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7123.438574] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7123.439751] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.451963] LustreError: Skipped 2 previous similar messages [ 7123.456474] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7123.467429] Lustre: Skipped 5 previous similar messages [ 7123.481114] Lustre: Skipped 11 previous similar messages [ 7128.886174] Lustre: server umount lustre-OST0001 complete [ 7133.664715] LustreError: 219980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.690490] LustreError: 219980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7136.569730] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7136.774636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7141.187788] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7141.361129] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7170.016551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.663803] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7175.678329] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7175.683160] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7175.698963] Lustre: Skipped 1 previous similar message [ 7176.299099] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7183.398943] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7184.838291] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7192.791675] Lustre: server umount lustre-OST0001 complete [ 7224.287427] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7224.525790] Lustre: server umount lustre-OST0000 complete [ 7234.974849] Lustre: server umount lustre-MDT0000 complete [ 7238.850734] LustreError: 220003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784192965 with bad export cookie 5219221004275931978 [ 7238.853554] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7238.866394] LustreError: 220003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7245.222996] Lustre: server umount lustre-MDT0001 complete [ 7270.786068] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:09:55 (1784192995) [ 7277.073077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7277.314643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7277.364890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7282.341871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7290.410400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.479474] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7290.504351] Lustre: 226778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7295.047069] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7302.744479] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7309.010873] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7316.569193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7316.772567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7323.232788] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7331.051366] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7341.771420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.832960] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7348.827935] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7357.092987] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7358.967527] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7364.226768] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7364.512440] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7369.621317] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7369.817518] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7376.599649] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff958f03e96000.ost_server_uuid 50 [ 7378.759806] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff958f03e96000.ost_server_uuid in IDLE state after 0 sec [ 7380.072843] Lustre: Permanently deactivating lustre-OST0001 [ 7380.092580] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7384.243043] Lustre: Permanently reactivating lustre-OST0001 [ 7384.257561] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7384.261992] Lustre: Skipped 2 previous similar messages [ 7390.181042] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7390.190599] LustreError: Skipped 3 previous similar messages [ 7390.198036] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.209891] Lustre: Skipped 6 previous similar messages [ 7390.232804] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7390.238524] Lustre: Skipped 12 previous similar messages [ 7394.971270] Lustre: server umount lustre-OST0001 complete [ 7413.215168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7413.356569] Lustre: server umount lustre-OST0000 complete [ 7423.209948] Lustre: server umount lustre-MDT0000 complete [ 7428.904715] LustreError: 227519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193155 with bad export cookie 5219221004275933259 [ 7428.929828] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.933301] LustreError: 227519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7429.407540] Lustre: server umount lustre-MDT0001 complete [ 7473.298615] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:13:17 (1784193197) [ 7482.199141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.472141] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7482.505391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7482.511854] Lustre: Skipped 2 previous similar messages [ 7487.423345] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7496.981300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.172030] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7497.222200] Lustre: 233476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7502.524403] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7510.855785] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7516.987889] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7524.919044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7525.203817] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7525.273196] Lustre: Permanently deactivating lustre-OST0000 [ 7525.294813] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7525.549396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.567634] Lustre: Skipped 10 previous similar messages [ 7532.288425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7533.672790] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7533.779920] LustreError: 235778:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7539.634626] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.566722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7544.612871] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7548.725146] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7552.105077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7553.451545] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7554.546753] Lustre: Permanently reactivating lustre-OST0000 [ 7557.198717] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7557.203864] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7557.216320] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7557.242383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7557.257977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7580.168448] Lustre: server umount lustre-OST0001 complete [ 7590.879377] Lustre: server umount lustre-OST0000 complete [ 7602.145283] Lustre: server umount lustre-MDT0000 complete [ 7606.380543] LustreError: 233470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193333 with bad export cookie 5219221004275934939 [ 7606.394830] LustreError: 233470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7606.411574] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7606.932981] Lustre: server umount lustre-MDT0001 complete [ 7621.614725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7625.063208] Key type lgssc unregistered [ 7625.381151] LNet: 238357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7625.385975] LNetError: 238357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7625.422612] LNet: Removed LNI 192.168.204.145@tcp [ 7626.580711] Key type .llcrypt unregistered [ 7626.585764] Key type ._llcrypt unregistered [ 7653.477714] Key type ._llcrypt registered [ 7653.481652] Key type .llcrypt registered [ 7653.624225] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7654.933835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7654.964651] alg: No test for adler32 (adler32-zlib) [ 7656.145527] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 7656.410784] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7658.087418] Key type lgssc registered [ 7659.329795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7670.792597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7670.811549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7672.399885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.620064] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7682.531101] LustreError: 239837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7682.562022] LustreError: 239837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7687.660734] LustreError: 239838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7687.921204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.052859] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7688.079859] Lustre: 239827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7688.087976] Lustre: Permanently deactivating lustre-MDT0001 [ 7688.108366] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7688.387958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7693.450662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7697.992570] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7701.997510] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7710.093288] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7718.985032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7719.510199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.206886] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7733.892353] LustreError: 241681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.916971] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7736.218997] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.374521] LustreError: 241681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.477047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7744.896334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.128298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.142957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7750.148453] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7752.388743] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7761.165226] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7765.680961] Lustre: Permanently reactivating lustre-MDT0001 [ 7765.726335] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7765.734676] Lustre: Skipped 2 previous similar messages [ 7767.263663] Lustre: lustre-MDT0001: Client 7d5d7f81-4bb0-4de2-90ec-1bd7ef91170c (at 192.168.204.45@tcp) reconnecting [ 7775.020327] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7775.282770] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7775.299564] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7775.310276] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7775.322153] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7777.093191] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7784.355480] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7784.499822] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7791.226863] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7791.512607] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7795.399230] Lustre: Permanently deactivating lustre-MDT0001 [ 7795.440834] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7795.452321] Lustre: Skipped 2 previous similar messages [ 7806.146024] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7816.675923] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.681817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.694238] Lustre: Skipped 1 previous similar message [ 7816.702422] Lustre: Skipped 2 previous similar messages [ 7816.837718] Lustre: server umount lustre-MDT0000 complete [ 7821.236458] LustreError: 239821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193548 with bad export cookie 4647355810686636914 [ 7821.238148] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7821.257536] LustreError: 239821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7821.515445] Lustre: server umount lustre-MDT0001 complete [ 7826.023746] Lustre: server umount lustre-OST0000 complete [ 7840.259976] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:19:25 (1784193565) [ 7852.676311] Lustre: server umount lustre-OST0001 complete [ 7860.468962] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7872.019509] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7915.507484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.957751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7915.971252] Lustre: Skipped 2 previous similar messages [ 7916.008990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7916.163252] Lustre: lustre-MDT0000: new disk, initializing [ 7916.294328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7916.364440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7923.396317] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7934.692177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.851899] Lustre: 248151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7934.902190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7934.908118] Lustre: Skipped 1 previous similar message [ 7935.023273] Lustre: lustre-MDT0001: new disk, initializing [ 7935.048960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7935.144902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7935.198324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7935.217146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7940.961048] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7951.122926] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7958.048785] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7966.176311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7966.433564] Lustre: lustre-OST0000: new disk, initializing [ 7966.440384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7966.463360] Lustre: 250119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7966.610080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7973.806233] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7974.493131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7974.510515] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7974.631814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7985.387393] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7988.080067] LustreError: 248156:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7998.165512] LustreError: 248156:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7998.198536] LustreError: 249375:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7998.594877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.835767] Lustre: lustre-OST0001: new disk, initializing [ 7998.839121] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7998.845405] Lustre: 251235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7998.944539] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.249764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8006.263280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8006.393466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8006.449647] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8008.223165] LustreError: 249375:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8016.294155] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8018.285505] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8020.730468] Lustre: lustre-OST0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 8021.986286] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8021.987048] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8021.998466] LustreError: Skipped 1 previous similar message [ 8022.013689] Lustre: Skipped 1 previous similar message [ 8025.061733] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8025.066706] Lustre: Skipped 3 previous similar messages [ 8026.644433] Lustre: server umount lustre-OST0001 complete [ 8030.186276] LustreError: 250511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8030.235937] LustreError: 250511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8040.424388] LustreError: 250104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.452884] LustreError: 250104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8045.539175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.543843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8045.558647] Lustre: Skipped 2 previous similar messages [ 8045.571754] Lustre: Skipped 2 previous similar messages [ 8051.488151] Lustre: server umount lustre-OST0000 complete [ 8055.777681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.782229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.786245] Lustre: Skipped 1 previous similar message [ 8055.789189] Lustre: Skipped 2 previous similar messages [ 8061.949336] Lustre: server umount lustre-MDT0000 complete [ 8066.015976] LustreError: 248141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784193792 with bad export cookie 4647355810686639728 [ 8066.016605] LustreError: 248161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8066.032556] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8066.039229] LustreError: 248141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8066.053466] LustreError: 248161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8080.351905] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8080.847270] Lustre: server umount lustre-MDT0001 complete [ 8094.242093] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8097.717150] Key type lgssc unregistered [ 8098.035675] LNet: 253272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8098.055050] LNetError: 253272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8098.084721] LNet: Removed LNI 192.168.204.145@tcp [ 8099.422743] Key type .llcrypt unregistered [ 8099.429281] Key type ._llcrypt unregistered [ 8148.709996] Key type ._llcrypt registered [ 8148.717287] Key type .llcrypt registered [ 8148.850777] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8150.149049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8150.306620] alg: No test for adler32 (adler32-zlib) [ 8151.596471] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 8151.900851] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8153.565645] Key type lgssc registered [ 8154.759970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.743494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.758387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.980590] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8165.016856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8165.163629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.522680] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8180.050256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.191562] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8180.233508] Lustre: 255479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8180.602975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8186.330854] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8196.121904] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8202.799295] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8210.572598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8210.902156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8211.146570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.787555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8218.606105] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8226.647208] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8279.011539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8279.023863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.050166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8279.524287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.557158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8284.145391] Lustre: server umount lustre-OST0000 complete [ 8289.016139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8304.510586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.761725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.513165] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8319.084721] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8319.098959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8319.100850] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8319.129663] Lustre: Skipped 1 previous similar message [ 8320.797792] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8365.537522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8365.543079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.555767] LustreError: Skipped 1 previous similar message [ 8365.556833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8365.581576] Lustre: Skipped 1 previous similar message [ 8367.142695] Lustre: server umount lustre-OST0000 complete [ 8375.269273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8375.282624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.313591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.316287] Lustre: Skipped 1 previous similar message [ 8377.807613] Lustre: server umount lustre-MDT0000 complete [ 8382.350848] LustreError: 255473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194109 with bad export cookie 7367838093189455531 [ 8382.357891] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8382.369244] LustreError: 255473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8382.879657] Lustre: server umount lustre-MDT0001 complete [ 8395.257699] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8398.807978] Key type lgssc unregistered [ 8399.193752] LNet: 260910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8399.203292] LNetError: 260910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8399.226041] LNet: Removed LNI 192.168.204.145@tcp [ 8400.435179] Key type .llcrypt unregistered [ 8400.441061] Key type ._llcrypt unregistered [ 8427.950800] Key type ._llcrypt registered [ 8427.954852] Key type .llcrypt registered [ 8428.130615] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8429.571875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8429.668046] alg: No test for adler32 (adler32-zlib) [ 8430.840970] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 8431.131470] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8432.839178] Key type lgssc registered [ 8434.119905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8443.520525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8443.531447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.219223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.885927] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8455.654774] LustreError: 262329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.681747] LustreError: 262329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8460.795521] LustreError: 262328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8461.740667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8462.296318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8467.747794] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8476.401138] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8482.959983] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8491.035659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.505657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8498.068207] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8505.773740] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8515.074930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8540.645074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8540.646498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.655937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8540.678148] Lustre: Skipped 1 previous similar message [ 8540.697283] Lustre: Skipped 1 previous similar message [ 8545.766719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8545.776469] Lustre: Skipped 1 previous similar message [ 8546.188534] Lustre: server umount lustre-OST0000 complete [ 8550.881522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.888513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.902187] Lustre: Skipped 1 previous similar message [ 8550.917237] Lustre: Skipped 1 previous similar message [ 8556.001942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.020116] Lustre: Skipped 1 previous similar message [ 8557.114555] Lustre: server umount lustre-MDT0000 complete [ 8559.008557] LustreError: 263057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.045613] LustreError: 263057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8561.356901] LustreError: 262310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194288 with bad export cookie 788086246801552681 [ 8561.361508] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8561.384048] LustreError: 262310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8561.776508] Lustre: server umount lustre-MDT0001 complete [ 8573.271956] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8576.464426] Key type lgssc unregistered [ 8576.814770] LNet: 266658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8576.822922] LNetError: 266658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8576.848142] LNet: Removed LNI 192.168.204.145@tcp [ 8577.713752] Key type .llcrypt unregistered [ 8577.715641] Key type ._llcrypt unregistered [ 8596.750817] Key type ._llcrypt registered [ 8596.752948] Key type .llcrypt registered [ 8596.863595] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8598.378236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8598.436072] alg: No test for adler32 (adler32-zlib) [ 8599.614660] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 8599.952502] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8601.615147] Key type lgssc registered [ 8603.254778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8611.633444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8611.669826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.202136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.347202] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8623.597121] LustreError: 267878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8623.628546] LustreError: 267878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8628.015175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.446643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8632.604804] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8640.470567] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8647.084283] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8654.168825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8654.530935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8658.617588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8660.936767] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8668.840140] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8689.648610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8689.654066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.665811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8689.666294] LustreError: Skipped 1 previous similar message [ 8689.702782] Lustre: Skipped 1 previous similar message [ 8690.969059] Lustre: server umount lustre-OST0000 complete [ 8699.879799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8699.892996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.900904] Lustre: Skipped 1 previous similar message [ 8699.926081] Lustre: Skipped 2 previous similar messages [ 8702.410732] Lustre: server umount lustre-MDT0000 complete [ 8704.993560] LustreError: 267878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.024165] LustreError: 267878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8706.408838] LustreError: 269723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194433 with bad export cookie 4892725746891385589 [ 8706.413182] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8706.420433] LustreError: 269723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8706.846723] Lustre: server umount lustre-MDT0001 complete [ 8718.697945] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8721.905873] Key type lgssc unregistered [ 8722.318568] LNet: 271559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8722.329267] LNetError: 271559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8722.350447] LNet: Removed LNI 192.168.204.145@tcp [ 8723.353279] Key type .llcrypt unregistered [ 8723.356793] Key type ._llcrypt unregistered [ 8749.599696] Key type ._llcrypt registered [ 8749.602148] Key type .llcrypt registered [ 8749.694234] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8750.742027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8750.813288] alg: No test for adler32 (adler32-zlib) [ 8752.199646] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 8752.588681] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8754.306195] Key type lgssc registered [ 8755.774278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8764.114146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8764.124229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.543525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.862445] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8775.142893] LustreError: 272978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.158020] LustreError: 272978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8778.646749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.063942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8783.832914] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8791.716780] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.881729] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8804.840785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8805.098313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8810.240508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8811.643578] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8820.286553] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8849.378246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8849.388136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.395425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.423111] Lustre: Skipped 1 previous similar message [ 8854.091585] Lustre: server umount lustre-OST0000 complete [ 8859.618591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8859.639740] Lustre: Skipped 1 previous similar message [ 8859.641689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.670035] Lustre: Skipped 2 previous similar messages [ 8864.737804] LustreError: 272978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8864.754503] LustreError: 272978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8864.855564] Lustre: server umount lustre-MDT0000 complete [ 8868.453373] LustreError: 272958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194595 with bad export cookie 13631411020765651748 [ 8868.460143] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.464369] LustreError: 272958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8868.846718] Lustre: server umount lustre-MDT0001 complete [ 8880.549356] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 8883.493862] Key type lgssc unregistered [ 8883.887409] LNet: 277284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8883.900057] LNetError: 277284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8883.912063] LNet: Removed LNI 192.168.204.145@tcp [ 8884.936312] Key type .llcrypt unregistered [ 8884.939234] Key type ._llcrypt unregistered [ 8904.980965] Key type ._llcrypt registered [ 8904.989376] Key type .llcrypt registered [ 8905.099299] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 8906.642031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8906.715941] alg: No test for adler32 (adler32-zlib) [ 8907.791836] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 8908.093102] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 8909.823223] Key type lgssc registered [ 8911.205886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8919.067306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8919.099352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.721088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8925.246199] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8930.284128] LustreError: 278522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8930.294724] LustreError: 278522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8935.394736] LustreError: 278524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.865540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.257175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8940.977267] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8949.157313] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8955.462984] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8963.160153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8963.475964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.494655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8970.128882] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 8977.392461] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8991.200811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.216073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.232058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8992.224324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8992.228803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8992.236807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.251305] Lustre: Skipped 1 previous similar message [ 8997.208671] Lustre: server umount lustre-OST0000 complete [ 9002.465198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.467365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.476543] Lustre: Skipped 1 previous similar message [ 9002.487582] Lustre: Skipped 1 previous similar message [ 9007.000176] Lustre: server umount lustre-MDT0000 complete [ 9007.585983] LustreError: 278523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.598773] LustreError: 278523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9011.122465] LustreError: 278487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194738 with bad export cookie 6311729449907324562 [ 9011.132566] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9011.136324] LustreError: 278487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9011.534805] Lustre: server umount lustre-MDT0001 complete [ 9023.026224] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9026.293691] Key type lgssc unregistered [ 9026.553644] LNet: 282226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9026.566247] LNetError: 282226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9026.586395] LNet: Removed LNI 192.168.204.145@tcp [ 9027.507964] Key type .llcrypt unregistered [ 9027.511648] Key type ._llcrypt unregistered [ 9039.452578] Key type ._llcrypt registered [ 9039.455808] Key type .llcrypt registered [ 9039.592358] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9060.011704] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9076.218715] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9077.571157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9077.608828] alg: No test for adler32 (adler32-zlib) [ 9078.724622] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 9078.940995] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9080.607258] Key type lgssc registered [ 9081.754388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9120.245271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9120.282562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.628645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9121.682956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9121.774320] Lustre: lustre-MDT0000: new disk, initializing [ 9121.971920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.023643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9128.377220] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9141.013637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.171851] Lustre: 285780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9141.225413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9141.229195] Lustre: Skipped 1 previous similar message [ 9141.289920] Lustre: lustre-MDT0001: new disk, initializing [ 9141.374758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9141.413692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9141.430967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9146.046481] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9156.514334] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9163.255996] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9170.812205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9171.118132] Lustre: lustre-OST0000: new disk, initializing [ 9171.122796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9171.132844] Lustre: 287752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9171.280328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.789712] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9180.752018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9180.778398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9180.932581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9188.081926] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9193.065620] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9193.293556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9198.734322] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9199.013414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9201.648131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.657653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9201.659033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9201.678356] Lustre: Skipped 1 previous similar message [ 9201.686724] LustreError: Skipped 1 previous similar message [ 9206.756745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9206.764964] Lustre: Skipped 2 previous similar messages [ 9207.634727] Lustre: server umount lustre-OST0000 complete [ 9212.897915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9212.911893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.954777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9216.993360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9216.997234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9218.321807] Lustre: server umount lustre-MDT0000 complete [ 9222.176080] LustreError: 286584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784194949 with bad export cookie 10675405602623193751 [ 9222.179511] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.187752] LustreError: 286584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9222.677056] Lustre: server umount lustre-MDT0001 complete [ 9232.325152] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:42:37 (1784194957) [ 9238.904913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.363202] LustreError: 290164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.381671] LustreError: 290164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9239.442991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9243.988792] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9244.641263] LustreError: 290165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.760366] LustreError: 290164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9252.291643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.904904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9257.412608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9264.995091] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9270.127571] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9276.698915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9277.081153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9282.429838] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9289.662489] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9299.941746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9299.953415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.962944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9299.976206] Lustre: Skipped 1 previous similar message [ 9299.991243] Lustre: Skipped 2 previous similar messages [ 9305.558673] Lustre: server umount lustre-OST0000 complete [ 9309.163368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9309.170901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.184915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9309.191572] Lustre: Skipped 2 previous similar messages [ 9315.305868] LustreError: 290164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.317768] LustreError: 290164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9315.368054] Lustre: server umount lustre-MDT0000 complete [ 9318.999466] LustreError: 290145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195045 with bad export cookie 10675405602623194626 [ 9319.004284] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9319.011959] LustreError: 290145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9319.256973] Lustre: server umount lustre-MDT0001 complete [ 9330.480430] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9333.281699] Key type lgssc unregistered [ 9333.566621] LNet: 293699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9333.577668] LNetError: 293699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9333.595291] LNet: Removed LNI 192.168.204.145@tcp [ 9334.334487] Key type .llcrypt unregistered [ 9334.337271] Key type ._llcrypt unregistered [ 9366.286987] Key type ._llcrypt registered [ 9366.289894] Key type .llcrypt registered [ 9366.406246] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9381.906890] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9383.081718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9383.191873] alg: No test for adler32 (adler32-zlib) [ 9384.449255] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 9384.786891] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9386.487237] Key type lgssc registered [ 9387.828477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9430.238966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9430.262625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.496122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9431.539649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9431.658834] Lustre: lustre-MDT0000: new disk, initializing [ 9431.713389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9431.727054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9435.771905] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9446.901051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.067788] Lustre: 297522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9447.161070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9447.167538] Lustre: Skipped 1 previous similar message [ 9447.174401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9447.253799] Lustre: lustre-MDT0001: new disk, initializing [ 9447.326162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9447.365181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9447.371386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9451.858246] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9460.992753] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9466.831902] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9473.316259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9473.718110] Lustre: lustre-OST2710: new disk, initializing [ 9473.720318] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9473.724708] Lustre: 299493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9473.799862] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9480.448243] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9480.830359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9480.843131] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9480.986931] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9491.658844] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9499.553561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9499.713990] Lustre: lustre-OST03e8: new disk, initializing [ 9499.722582] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9499.734457] Lustre: 300557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9499.922701] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9505.238791] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9508.423227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9508.429630] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9508.502184] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9514.097225] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9521.853271] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9522.093714] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9527.164222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9527.395940] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9532.455333] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9532.779843] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9537.816587] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9538.022517] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9544.673308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9544.676519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9544.692092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9544.702898] Lustre: Skipped 1 previous similar message [ 9545.279311] Lustre: server umount lustre-MDT0000 complete [ 9549.129563] LustreError: 297516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195276 with bad export cookie 16383000576745471595 [ 9549.136118] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9549.145788] LustreError: 297516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9549.498308] Lustre: server umount lustre-MDT0001 complete [ 9563.285826] Lustre: server umount lustre-OST2710 complete [ 9565.160960] Lustre: 295323:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784195276/real 1784195276] req@0000000000a69722 x1870864172780160/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784195292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9565.203540] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.216286] Lustre: Skipped 2 previous similar messages [ 9567.330728] Lustre: server umount lustre-OST03e8 complete [ 9586.088126] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 9597.635447] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9637.463627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9637.742552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9637.785882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9637.869695] Lustre: lustre-MDT0000: new disk, initializing [ 9638.011149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9638.023702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9642.758760] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9653.140319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.236947] Lustre: 305810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9653.288863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9653.293796] Lustre: Skipped 1 previous similar message [ 9653.353331] Lustre: lustre-MDT0001: new disk, initializing [ 9653.468458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9653.531348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9653.543648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9658.251219] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9668.688684] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9675.071852] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9681.830386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.109551] Lustre: lustre-OST0000: new disk, initializing [ 9682.114255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9682.119706] Lustre: 307779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9682.209457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9689.068448] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9691.186501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9691.205899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9691.264447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9699.404609] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9704.850945] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9705.165378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9709.940779] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9710.218427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9716.706218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.715770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9716.737149] Lustre: Skipped 1 previous similar message [ 9716.755044] Lustre: Skipped 3 previous similar messages [ 9718.480271] Lustre: server umount lustre-OST0000 complete [ 9725.409729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9725.414105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9725.433859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9725.441231] Lustre: Skipped 1 previous similar message [ 9732.069633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9732.073833] Lustre: Skipped 3 previous similar messages [ 9737.183321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9737.558812] Lustre: server umount lustre-MDT0000 complete [ 9742.460793] LustreError: 305802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195469 with bad export cookie 16383000576745473275 [ 9742.474247] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9742.490223] LustreError: 305802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9743.132190] Lustre: server umount lustre-MDT0001 complete [ 9753.650415] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:51:18 (1784195478) [ 9754.970063] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9756.799906] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:51:21 (1784195481) [ 9760.133219] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9788.000525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.272502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9788.293438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9788.478858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.742906] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9803.122714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.275401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9803.309735] Lustre: 311639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9808.808630] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9831.810799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9832.022119] Lustre: Denying initial registration attempt from nid 192.168.204.145@tcp, specified as failover [ 9832.027364] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9832.032758] LustreError: 313093:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9832.037388] LustreError: 313093:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9832.041920] LustreError: 313093:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9832.099670] Lustre: server umount lustre-OST0000 complete [ 9832.102421] LustreError: 313093:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9839.585218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9839.600658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9839.609693] Lustre: Skipped 1 previous similar message [ 9839.616474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9839.623887] Lustre: Skipped 2 previous similar messages [ 9845.612544] Lustre: server umount lustre-MDT0000 complete [ 9846.756729] LustreError: 311647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9846.786788] LustreError: 311647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9850.558681] LustreError: 311632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195577 with bad export cookie 16383000576745474157 [ 9850.571456] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9850.576515] LustreError: 311632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9850.977231] Lustre: server umount lustre-MDT0001 complete [ 9865.066776] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 9869.051676] Key type lgssc unregistered [ 9869.495784] LNet: 314215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9869.501974] LNetError: 314215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9870.564989] LNet: Removed LNI 192.168.204.145@tcp [ 9871.653108] Key type .llcrypt unregistered [ 9871.656754] Key type ._llcrypt unregistered [ 9885.826079] Key type ._llcrypt registered [ 9885.829522] Key type .llcrypt registered [ 9885.943529] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9887.039741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9887.082897] alg: No test for adler32 (adler32-zlib) [ 9888.248061] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [ 9888.538789] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 9890.233704] Key type lgssc registered [ 9891.388050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9942.849272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 9951.860468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9951.888315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.128186] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9953.150655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9953.315487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9958.458944] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9967.210461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.317970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9967.349917] Lustre: 316983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9967.631364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9972.143123] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 9979.785248] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9985.787644] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10006.395796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10006.743764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10007.060181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10014.002163] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10022.399840] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10030.051553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.060387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10030.087134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.103670] LustreError: Skipped 1 previous similar message [10035.178039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10035.214034] Lustre: Skipped 2 previous similar messages [10039.263256] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10039.487583] Lustre: server umount lustre-OST0000 complete [10044.385413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10044.394504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.409983] Lustre: Skipped 1 previous similar message [10044.424370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10045.420488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10050.529767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10050.537054] Lustre: Skipped 3 previous similar messages [10055.649122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10055.658937] Lustre: Skipped 1 previous similar message [10058.207207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10058.377691] Lustre: server umount lustre-MDT0000 complete [10063.023295] LustreError: 317725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195789 with bad export cookie 6548351669215646180 [10063.031184] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.037063] LustreError: 317725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10063.405775] Lustre: server umount lustre-MDT0001 complete [10075.358243] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10078.581791] Key type lgssc unregistered [10078.895989] LNet: 320560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10078.906690] LNetError: 320560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10078.926285] LNet: Removed LNI 192.168.204.145@tcp [10079.976124] Key type .llcrypt unregistered [10079.977597] Key type ._llcrypt unregistered [10106.111129] Key type ._llcrypt registered [10106.116170] Key type .llcrypt registered [10106.230341] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10107.088154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10107.230567] alg: No test for adler32 (adler32-zlib) [10108.228450] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [10108.452869] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10110.119436] Key type lgssc registered [10111.159837] Lustre: Echo OBD driver; http://www.lustre.org/ [10118.322396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10118.353714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10119.857294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10124.415701] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10129.380934] LustreError: 321978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.398478] LustreError: 321978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10132.282188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.603390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10136.955941] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10145.848135] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10152.501857] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10162.475099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10162.858479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10168.554293] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10177.240561] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10187.747086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.757920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10187.761824] Lustre: Skipped 2 previous similar messages [10187.769123] Lustre: Skipped 2 previous similar messages [10190.071704] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [10192.444782] Lustre: server umount lustre-MDT0000 complete [10192.866400] LustreError: 321977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.883760] LustreError: 321977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10195.151143] LustreError: 323936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.340761] LustreError: 322713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784195923 with bad export cookie 10193151162363036873 [10196.351160] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10197.984832] LustreError: 321972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.989570] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.026252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10200.290091] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [10202.018642] LustreError: 321973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10202.051209] LustreError: 321973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10202.726663] Lustre: server umount lustre-MDT0001 complete [10210.919224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10221.062110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.834811] LustreError: 323810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.920819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10221.978587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10226.620799] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10231.173715] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10235.212732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.591076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10239.110787] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10241.009235] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10246.372914] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10247.779386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10253.561725] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10255.332535] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10261.475255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10261.478205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10261.496032] Lustre: Skipped 1 previous similar message [10261.500815] Lustre: Skipped 1 previous similar message [10261.508423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10266.388596] Lustre: server umount lustre-OST0000 complete [10271.712397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10271.717715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10271.730641] Lustre: Skipped 1 previous similar message [10276.300227] Lustre: server umount lustre-MDT0000 complete [10276.834300] LustreError: 325575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.853146] LustreError: 325575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10280.597516] LustreError: 321958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196007 with bad export cookie 10193151162363044391 [10280.604202] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.614062] LustreError: 321958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10280.950519] Lustre: server umount lustre-MDT0001 complete [10291.987215] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10294.844949] Key type lgssc unregistered [10295.051271] LNet: 328248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10295.055704] LNetError: 328248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10295.078419] LNet: Removed LNI 192.168.204.145@tcp [10295.777984] Key type .llcrypt unregistered [10295.784085] Key type ._llcrypt unregistered [10321.434914] Key type ._llcrypt registered [10321.439177] Key type .llcrypt registered [10321.605945] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10322.835530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10322.884306] alg: No test for adler32 (adler32-zlib) [10323.968123] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [10324.231691] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10325.967497] Key type lgssc registered [10327.093219] Lustre: Echo OBD driver; http://www.lustre.org/ [10335.351538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10335.380203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.881628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10341.826905] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10346.979049] LustreError: 329677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.010077] LustreError: 329677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10350.515874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10351.033941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10355.896478] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10363.115629] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10367.905705] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10370.506122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.516531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10370.521270] Lustre: Skipped 1 previous similar message [10370.531660] Lustre: Skipped 1 previous similar message [10375.650235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.660103] Lustre: Skipped 1 previous similar message [10376.829773] Lustre: server umount lustre-MDT0000 complete [10379.942385] LustreError: 329657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196106 with bad export cookie 8842592931515249160 [10379.947389] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.952812] LustreError: 329657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10380.217194] Lustre: server umount lustre-MDT0001 complete [10395.189779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10395.413468] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10395.447246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10395.612293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10400.533737] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10409.554276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10409.753247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10409.807180] Lustre: 332263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10410.128318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10415.292882] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10424.846913] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10430.953511] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10440.842995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10441.140217] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10441.159332] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10441.173623] LustreError: 334149:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10441.186499] LustreError: 334149:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10441.193307] LustreError: 334149:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10441.295816] Lustre: server umount lustre-OST0000 complete [10441.301222] LustreError: 334149:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10451.385424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10451.586067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10451.799147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10457.681875] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10460.781496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10464.717877] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10473.011505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.143461] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10473.267773] Lustre: lustre-OST0001: new disk, initializing [10473.271589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10473.276557] Lustre: 335689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10473.392962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10479.192529] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10481.872849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10481.901565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10482.018653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10489.349792] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10497.504574] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10497.511377] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10497.522581] LustreError: Skipped 1 previous similar message [10497.533559] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10497.542580] Lustre: Skipped 1 previous similar message [10497.563560] Lustre: Skipped 1 previous similar message [10498.703390] Lustre: server umount lustre-OST0001 complete [10502.636920] LustreError: 334616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.659803] LustreError: 334616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10506.722569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10506.748167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.775488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10507.745778] LustreError: 334618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.788197] LustreError: 334618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10508.837260] Lustre: server umount lustre-OST0000 complete [10517.484142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10517.491817] LustreError: Skipped 1 previous similar message [10517.499488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10517.511548] Lustre: Skipped 1 previous similar message [10517.517074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.521884] Lustre: Skipped 2 previous similar messages [10528.224783] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10528.226038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.260350] Lustre: Skipped 4 previous similar messages [10528.950150] Lustre: server umount lustre-MDT0000 complete [10532.340487] LustreError: 333369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.378542] LustreError: 333369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10533.589702] LustreError: 332256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196260 with bad export cookie 8842592931515249713 [10533.596212] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10533.605138] LustreError: 332256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10533.863476] Lustre: server umount lustre-MDT0001 complete [10545.022342] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [10547.642320] Key type lgssc unregistered [10547.976436] LNet: 337651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10547.990520] LNetError: 337651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10548.024571] LNet: Removed LNI 192.168.204.145@tcp [10548.822501] Key type .llcrypt unregistered [10548.832149] Key type ._llcrypt unregistered [10627.763567] Key type ._llcrypt registered [10627.765980] Key type .llcrypt registered [10627.868756] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [10644.302345] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [10645.541148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10645.663472] alg: No test for adler32 (adler32-zlib) [10646.751500] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [10647.018613] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [10648.759166] Key type lgssc registered [10650.057243] Lustre: Echo OBD driver; http://www.lustre.org/ [10687.768490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10687.785584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.095815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10689.150780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10689.250380] Lustre: lustre-MDT0000: new disk, initializing [10689.313537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.332083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10693.811766] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10704.683032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10704.820282] Lustre: 342588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10704.890244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10704.898781] Lustre: Skipped 1 previous similar message [10705.019365] Lustre: lustre-MDT0001: new disk, initializing [10705.122920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10705.159128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10705.162883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10709.780814] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10721.970760] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10729.756209] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10737.175985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10737.539656] Lustre: lustre-OST0000: new disk, initializing [10737.545947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10737.558666] Lustre: 344558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10737.668514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10743.392686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10743.403500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10743.512704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10744.174857] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10753.344861] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10758.070325] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10758.295787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10762.764174] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10762.974885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10769.379686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10769.386962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10769.393507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10769.399870] Lustre: Skipped 1 previous similar message [10774.521849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10774.535730] Lustre: Skipped 2 previous similar messages [10779.103137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10779.256632] Lustre: server umount lustre-OST0000 complete [10784.742418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.748318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10784.778152] Lustre: Skipped 1 previous similar message [10784.793020] Lustre: Skipped 1 previous similar message [10789.534343] Lustre: server umount lustre-MDT0000 complete [10793.359050] LustreError: 342580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196520 with bad export cookie 8470821086963318407 [10793.361590] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10793.371359] LustreError: 342580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10793.670569] Lustre: server umount lustre-MDT0001 complete [10804.060292] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:08:48 (1784196528) [10812.612652] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:08:57 (1784196537) [10822.396288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.898914] LustreError: 347315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10822.914865] LustreError: 347315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10822.963265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10828.260749] LustreError: 347316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10828.556911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10832.358108] LustreError: 347315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10837.474663] LustreError: 347316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.035084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.518717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10844.681587] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10853.214727] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10859.824360] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10866.909238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10867.178763] LustreError: 349136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.389291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10874.685086] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10883.506245] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10891.943841] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10894.124615] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10895.878566] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10898.096757] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10900.294473] Lustre: Failing over lustre-MDT0000 [10900.547774] Lustre: server umount lustre-MDT0000 complete [10900.964894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10900.982052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.013051] LustreError: 347316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.555512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10904.567987] Lustre: Skipped 1 previous similar message [10917.077883] LustreError: 348411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.093415] LustreError: 348411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [10920.927186] Lustre: 340813:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784196631/real 1784196631] req@000000007f41a1ea x1870865496090752/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784196647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10920.972422] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10921.816331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.209564] LustreError: 340810:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000022ab8d0 x1870865496096128/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10930.630703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10930.751340] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10933.449498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10935.801949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10935.857225] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10935.924783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10936.718139] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10948.837328] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10950.484580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10954.528672] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10956.658636] Lustre: Failing over lustre-MDT0000 [10957.114825] Lustre: server umount lustre-MDT0000 complete [10959.077112] LustreError: 348411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10959.094096] LustreError: 348411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10961.383182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10961.386543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10977.589189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10977.761156] Lustre: 340814:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784196688/real 1784196688] req@0000000098595035 x1870865496113536/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784196704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10977.773399] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10978.259441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10978.354171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10980.557984] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10983.405080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10983.423168] Lustre: Skipped 2 previous similar messages [10983.474347] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10983.557975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10984.243748] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [10995.146597] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10997.351699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11008.997228] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11008.998692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11009.003721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11009.022887] Lustre: Skipped 4 previous similar messages [11009.032541] Lustre: Skipped 1 previous similar message [11010.351155] Lustre: server umount lustre-MDT0000 complete [11014.469625] LustreError: 347296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196741 with bad export cookie 8470821086963321116 [11014.494257] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11014.950428] Lustre: server umount lustre-MDT0001 complete [11030.156927] Lustre: server umount lustre-OST0000 complete [11047.107351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.642352] LustreError: 353659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.656700] LustreError: 353659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [11047.744580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11052.799055] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11062.191543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.288855] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11075.157123] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11081.328170] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11088.207544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11092.676883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11095.117219] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11102.279252] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11113.453634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11113.463201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11113.483466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11113.488121] Lustre: Skipped 2 previous similar messages [11114.507128] Lustre: server umount lustre-OST0000 complete [11123.683711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11123.698566] Lustre: Skipped 3 previous similar messages [11124.523558] Lustre: server umount lustre-MDT0000 complete [11128.591201] LustreError: 353639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196855 with bad export cookie 8470821086963321858 [11128.601209] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.605364] LustreError: 353639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11128.959087] Lustre: server umount lustre-MDT0001 complete [11140.791654] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11144.115440] Key type lgssc unregistered [11144.414883] LNet: 357192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11144.425501] LNetError: 357192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11144.453433] LNet: Removed LNI 192.168.204.145@tcp [11145.452222] Key type .llcrypt unregistered [11145.455591] Key type ._llcrypt unregistered [11173.973308] Key type ._llcrypt registered [11173.976335] Key type .llcrypt registered [11174.106616] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11175.237556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11175.314469] alg: No test for adler32 (adler32-zlib) [11176.520751] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [11176.820788] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11178.495206] Key type lgssc registered [11179.690251] Lustre: Echo OBD driver; http://www.lustre.org/ [11187.338113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11187.376554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.843203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.063799] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11198.439554] LustreError: 358710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11198.457577] LustreError: 358710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11202.476558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.842356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11207.078561] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11214.969096] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11221.099204] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11228.428793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11228.932991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11234.027378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11235.803562] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11243.480456] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11259.360532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11259.369101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.385843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11260.627634] Lustre: server umount lustre-MDT0000 complete [11262.952745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.955844] LustreError: 358710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.969549] Lustre: Skipped 1 previous similar message [11262.984946] LustreError: 358710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11263.871875] LustreError: 358690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784196990 with bad export cookie 1977902572363912624 [11263.896313] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11264.100436] Lustre: server umount lustre-MDT0001 complete [11277.988978] Lustre: server umount lustre-OST0000 complete [11290.918785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.287349] LustreError: 362439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.332743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11294.846211] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11296.737903] LustreError: 362440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.629948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.874288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11305.575833] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11311.983411] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11317.844199] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11323.702264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11323.996737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11328.916488] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11335.576831] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11337.343244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11353.062388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11353.063544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.074747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11353.084102] Lustre: Skipped 2 previous similar messages [11353.093673] Lustre: Skipped 2 previous similar messages [11357.946806] Lustre: server umount lustre-MDT0000 complete [11358.177915] LustreError: 362440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.192072] LustreError: 362440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11361.227747] LustreError: 362420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197088 with bad export cookie 1977902572363913737 [11361.244848] LustreError: 362420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11361.255098] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11361.464560] Lustre: server umount lustre-MDT0001 complete [11374.354167] Lustre: server umount lustre-OST0000 complete [11389.930106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11390.304434] LustreError: 366349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.408544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11394.159273] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11402.374824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.525040] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11413.323691] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11418.916736] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11425.673641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11426.021255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11426.025925] Lustre: Skipped 1 previous similar message [11431.589479] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11436.273284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11437.684355] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11441.636236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11441.639646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11441.644439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11441.652340] Lustre: Skipped 1 previous similar message [11443.577401] Lustre: server umount lustre-OST0000 complete [11446.753347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11446.757950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11446.770586] Lustre: Skipped 1 previous similar message [11446.775596] Lustre: Skipped 1 previous similar message [11451.872637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11451.887126] Lustre: Skipped 2 previous similar messages [11452.751742] Lustre: server umount lustre-MDT0000 complete [11455.457927] LustreError: 366331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197182 with bad export cookie 1977902572363914759 [11455.459581] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11455.467476] LustreError: 366331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11455.793077] Lustre: server umount lustre-MDT0001 complete [11464.684154] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11466.783869] Key type lgssc unregistered [11467.027307] LNet: 369878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11467.037597] LNetError: 369878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11468.072301] LNet: Removed LNI 192.168.204.145@tcp [11468.814841] Key type .llcrypt unregistered [11468.818523] Key type ._llcrypt unregistered [11489.663517] Key type ._llcrypt registered [11489.666310] Key type .llcrypt registered [11489.775366] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11490.628733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11490.654938] alg: No test for adler32 (adler32-zlib) [11491.619883] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [11491.821202] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11493.480282] Key type lgssc registered [11494.358382] Lustre: Echo OBD driver; http://www.lustre.org/ [11500.127549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11500.142924] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11500.149798] LustreError: 371303:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11500.171261] LustreError: 371303:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11500.179609] LustreError: 371303:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11500.187231] LustreError: 371303:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11500.195318] LustreError: 371303:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11506.425452] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11506.435297] LustreError: 371700:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11506.460661] LustreError: 371700:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11506.467107] LustreError: 371700:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11506.474562] LustreError: 371700:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11506.481671] LustreError: 371700:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11520.890713] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11522.760324] Key type lgssc unregistered [11522.949704] LNet: 372705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11522.958084] LNetError: 372705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11522.980645] LNet: Removed LNI 192.168.204.145@tcp [11523.575698] Key type .llcrypt unregistered [11523.578672] Key type ._llcrypt unregistered [11538.620727] Key type ._llcrypt registered [11538.623562] Key type .llcrypt registered [11538.709425] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [11552.390566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11553.526426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11553.656206] alg: No test for adler32 (adler32-zlib) [11554.657983] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [11554.853922] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11556.503161] Key type lgssc registered [11557.338140] Lustre: Echo OBD driver; http://www.lustre.org/ [11588.784347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11588.812196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.005674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11590.025822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11590.075363] Lustre: lustre-MDT0000: new disk, initializing [11590.139612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11590.155442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11593.190993] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11601.774345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.927725] Lustre: 375861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11602.000873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11602.005114] Lustre: Skipped 1 previous similar message [11602.098880] Lustre: lustre-MDT0001: new disk, initializing [11602.211213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11602.263269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11602.276412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11605.918614] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11614.313261] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11619.395936] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11625.587389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11625.815173] Lustre: lustre-OST0000: new disk, initializing [11625.819642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11625.826596] Lustre: 377829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11625.916389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11631.871393] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11632.695901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11632.708152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11632.909628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11640.459519] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11644.112712] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11644.279860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11647.962386] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11648.152451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11653.619418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11653.621101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11653.644599] LustreError: Skipped 1 previous similar message [11653.649158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11653.657064] Lustre: Skipped 1 previous similar message [11655.766278] Lustre: server umount lustre-OST0000 complete [11663.841061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11663.841557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.848226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11663.879171] Lustre: Skipped 2 previous similar messages [11665.196191] Lustre: server umount lustre-MDT0000 complete [11668.447306] LustreError: 375853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197395 with bad export cookie 2031717381984676901 [11668.455598] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11668.455717] LustreError: 375853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11668.718438] Lustre: server umount lustre-MDT0001 complete [11676.021894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11677.569772] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:23:22 (1784197402) [11683.163346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.660492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11687.725172] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11688.928805] LustreError: 380289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.946416] LustreError: 380289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11694.049530] LustreError: 380288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.844088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.134638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11698.829888] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11705.737752] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11710.969435] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11716.439455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11716.694672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11721.136453] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11727.472427] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11733.913408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11734.059321] Lustre: lustre-OST0001: new disk, initializing [11734.062701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11734.067733] Lustre: 383121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11734.133614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11739.874946] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11741.268848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11741.274488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11741.341125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11747.194703] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11751.916614] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11751.918551] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11751.933940] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11751.942461] Lustre: Skipped 1 previous similar message [11751.951756] Lustre: Skipped 2 previous similar messages [11756.492551] Lustre: server umount lustre-OST0001 complete [11757.033395] LustreError: 382109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11757.055994] LustreError: 382109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11762.157111] LustreError: 383952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.171799] LustreError: 383952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11767.273687] LustreError: 382110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.288420] LustreError: 382110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11776.482460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11776.492270] LustreError: Skipped 1 previous similar message [11776.497271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11776.518950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11776.523993] Lustre: Skipped 1 previous similar message [11777.513744] LustreError: 383586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.528112] LustreError: 383586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11779.220549] Lustre: server umount lustre-OST0000 complete [11782.625706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11782.637530] Lustre: Skipped 1 previous similar message [11782.647602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11782.652030] Lustre: Skipped 2 previous similar messages [11787.233451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11788.854864] Lustre: server umount lustre-MDT0000 complete [11792.430993] LustreError: 380269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197519 with bad export cookie 2031717381984677776 [11792.445467] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11792.447331] LustreError: 380269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11798.979638] Lustre: server umount lustre-MDT0001 complete [11809.871932] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [11813.035186] Key type lgssc unregistered [11813.355200] LNet: 385110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11813.360270] LNetError: 385110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11813.372444] LNet: Removed LNI 192.168.204.145@tcp [11814.418481] Key type .llcrypt unregistered [11814.423606] Key type ._llcrypt unregistered [11862.763908] Key type ._llcrypt registered [11862.767242] Key type .llcrypt registered [11862.877928] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [11863.971568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11863.985667] alg: No test for adler32 (adler32-zlib) [11865.085454] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [11865.316487] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [11866.983279] Key type lgssc registered [11867.942928] Lustre: Echo OBD driver; http://www.lustre.org/ [11875.441214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11875.470331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.744556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11876.763387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11876.955526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11880.886333] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11888.402320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.527332] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11888.573953] Lustre: 387910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11888.814290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11892.523468] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11899.502739] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11904.398681] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11908.064608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.078576] Lustre: Skipped 1 previous similar message [11908.079707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11908.092437] Lustre: Skipped 1 previous similar message [11912.520251] Lustre: server umount lustre-MDT0000 complete [11916.436956] LustreError: 388653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197643 with bad export cookie 18104949962670091394 [11916.445112] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11916.454518] LustreError: 388653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11916.782785] Lustre: server umount lustre-MDT0001 complete [11925.568583] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:27:30 (1784197650) [11931.574722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.124389] LustreError: 390423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.146938] LustreError: 390423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11932.199547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11935.951415] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11937.252948] LustreError: 390424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.370789] LustreError: 390423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.013709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.411683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11947.663544] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11954.646691] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11959.254933] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11965.120397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11965.290218] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11965.478740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11970.596318] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [11977.541588] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11984.921248] Lustre: Permanently deactivating lustre-OST0000 [11984.935708] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11993.106670] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11994.225384] LustreError: 393408:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11994.238992] LustreError: 393408:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11994.245792] LustreError: 393408:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11997.230265] LustreError: 393457:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11997.240126] LustreError: 393457:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12005.187098] Lustre: server umount lustre-OST0000 complete [12007.527918] LustreError: 393707:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12007.534345] LustreError: 393707:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12009.952384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12009.972513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12010.012090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.001598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.006741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.023769] Lustre: Skipped 1 previous similar message [12017.122622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12017.128983] Lustre: Skipped 1 previous similar message [12019.241619] Lustre: server umount lustre-MDT0000 complete [12022.247957] LustreError: 390424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.272283] LustreError: 390424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12022.664913] LustreError: 390403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197749 with bad export cookie 18104949962670091975 [12022.674073] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12022.685087] LustreError: 390403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12033.169960] Lustre: server umount lustre-MDT0001 complete [12041.217692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12044.911856] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12047.243830] LustreError: 394865:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12052.616943] Lustre: 395203:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.145@tcp [12054.333888] Lustre: server umount MGS complete [12064.911108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12065.307522] LustreError: 395927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.329996] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12065.334651] Lustre: Skipped 1 previous similar message [12065.383217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12069.386679] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12075.496927] LustreError: 395927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12075.514534] LustreError: 395927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12076.851199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.045338] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12080.775476] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12087.258338] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12092.439849] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12094.113359] Lustre: Permanently reactivating lustre-OST0000 [12094.124503] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12094.131839] Lustre: Skipped 2 previous similar messages [12123.615116] LustreError: 397756:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12129.217975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12129.566608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.577459] Lustre: Skipped 1 previous similar message [12134.576532] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12141.431485] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12149.219547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.222629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12149.250293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.258996] Lustre: Skipped 1 previous similar message [12151.663826] Lustre: server umount lustre-OST0000 complete [12158.945444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12158.952676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.962916] Lustre: Skipped 1 previous similar message [12158.966749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.183194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12169.323932] Lustre: server umount lustre-MDT0000 complete [12169.699207] LustreError: 395928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.715200] LustreError: 395928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12172.126731] LustreError: 397500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784197899 with bad export cookie 18104949962670093025 [12172.132723] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.136513] LustreError: 397500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12172.389400] Lustre: server umount lustre-MDT0001 complete [12180.553856] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12182.685527] Key type lgssc unregistered [12182.959912] LNet: 400436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12182.964844] LNetError: 400436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12182.977270] LNet: Removed LNI 192.168.204.145@tcp [12183.597759] Key type .llcrypt unregistered [12183.603593] Key type ._llcrypt unregistered [12198.674439] Key type ._llcrypt registered [12198.677873] Key type .llcrypt registered [12198.763731] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [12210.652580] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12211.467671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12211.491772] alg: No test for adler32 (adler32-zlib) [12212.453642] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [12212.659120] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12214.335194] Key type lgssc registered [12215.162655] Lustre: Echo OBD driver; http://www.lustre.org/ [12243.511447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12243.563061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12244.781543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12244.820111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12244.871692] Lustre: lustre-MDT0000: new disk, initializing [12244.959084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12244.974403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12249.033871] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12258.866757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.969995] Lustre: 403571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12259.012431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12259.015758] Lustre: Skipped 1 previous similar message [12259.092435] Lustre: lustre-MDT0001: new disk, initializing [12259.197446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12259.226759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12259.243696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12262.888724] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12270.637743] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12275.679331] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12280.953546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12281.131168] Lustre: lustre-OST0000: new disk, initializing [12281.134390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12281.139317] Lustre: 405540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12281.203885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12286.069286] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12287.544479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12287.550805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12287.620645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12293.772946] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12297.199355] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12297.336498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12301.123228] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12301.286163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12303.338467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12303.340203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12303.344543] LustreError: Skipped 1 previous similar message [12303.345394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12303.373545] Lustre: Skipped 1 previous similar message [12308.456482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12308.459727] Lustre: Skipped 2 previous similar messages [12313.570685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12313.577486] Lustre: Skipped 1 previous similar message [12317.151116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12317.292447] Lustre: server umount lustre-OST0000 complete [12320.736928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12320.746821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.760787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.813217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12326.555146] Lustre: server umount lustre-MDT0000 complete [12329.213798] LustreError: 404375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198056 with bad export cookie 7337699942234706409 [12329.220814] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.224907] LustreError: 404375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12329.542929] Lustre: server umount lustre-MDT0001 complete [12336.631273] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12337.931769] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:34:23 (1784198063) [12343.133227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.433786] LustreError: 407998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.455060] LustreError: 407998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12343.498603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12346.679637] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12348.896604] LustreError: 407999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.459773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12353.775037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12357.251840] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12362.945957] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12367.273853] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12372.659495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12373.058573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12378.571272] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12384.864886] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12386.552987] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12393.374232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12398.570348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12398.578396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.599340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12398.603469] Lustre: Skipped 2 previous similar messages [12403.957143] Lustre: server umount lustre-OST0000 complete [12409.826033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.828654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12409.846161] Lustre: Skipped 2 previous similar messages [12409.865962] Lustre: Skipped 3 previous similar messages [12413.435466] Lustre: server umount lustre-MDT0000 complete [12414.944749] LustreError: 407998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12414.962317] LustreError: 407998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12416.316574] LustreError: 410972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198143 with bad export cookie 7337699942234707291 [12416.319627] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12416.323609] LustreError: 410972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12416.576901] Lustre: server umount lustre-MDT0001 complete [12426.200121] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12428.620220] Key type lgssc unregistered [12428.851720] LNet: 411723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12428.867910] LNetError: 411723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12428.882775] LNet: Removed LNI 192.168.204.145@tcp [12429.741774] Key type .llcrypt unregistered [12429.746318] Key type ._llcrypt unregistered [12449.501760] Key type ._llcrypt registered [12449.503892] Key type .llcrypt registered [12449.579444] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12449.777479] Key type .llcrypt unregistered [12449.783506] Key type ._llcrypt unregistered [12463.879550] Key type ._llcrypt registered [12463.881622] Key type .llcrypt registered [12463.970592] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12464.812744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12464.942734] alg: No test for adler32 (adler32-zlib) [12466.026847] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [12466.248331] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12467.895163] Key type lgssc registered [12468.576364] Lustre: Echo OBD driver; http://www.lustre.org/ [12473.773843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12473.786062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12475.072744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12477.680828] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12482.941762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12483.170339] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12483.180882] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12483.186846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12485.218991] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12485.226103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12485.227140] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12486.656151] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12490.723605] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.737882] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12491.024894] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12494.735128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12494.932301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12496.816962] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12515.299269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.302860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12515.303737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12515.305555] Lustre: Skipped 1 previous similar message [12520.418175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12520.421591] Lustre: Skipped 1 previous similar message [12520.572541] Lustre: server umount lustre-OST0000 complete [12525.545373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12525.548228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12525.554278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12525.569109] Lustre: Skipped 1 previous similar message [12525.573833] Lustre: Skipped 2 previous similar messages [12529.691714] Lustre: server umount lustre-MDT0000 complete [12530.656947] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.670272] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12533.313618] LustreError: 413895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198260 with bad export cookie 195022085390206896 [12533.317321] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12533.339703] LustreError: 413895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12533.828358] Lustre: server umount lustre-MDT0001 complete [12544.765686] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12547.073949] Key type lgssc unregistered [12547.316706] LNet: 417077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12547.324651] LNetError: 417077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12547.334711] LNet: Removed LNI 192.168.204.145@tcp [12547.968896] Key type .llcrypt unregistered [12547.973073] Key type ._llcrypt unregistered [12567.752550] Key type ._llcrypt registered [12567.755151] Key type .llcrypt registered [12567.837222] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12568.592401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12568.707457] alg: No test for adler32 (adler32-zlib) [12569.721890] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [12569.947300] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12571.631170] Key type lgssc registered [12572.537719] Lustre: Echo OBD driver; http://www.lustre.org/ [12578.564269] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12578.579371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12594.143519] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12599.263279] LustreError: 418447:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [12625.951228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12632.405330] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12639.914790] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12646.146418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12646.617526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12646.735459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12650.402261] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12652.003838] LustreError: 418472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.021522] LustreError: 418472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12657.124442] LustreError: 418473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12658.050078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12658.282773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12661.704776] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12668.586608] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12673.470874] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12681.186871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12681.189334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12681.202480] Lustre: Skipped 1 previous similar message [12682.470504] Lustre: server umount lustre-OST0000 complete [12686.304651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12686.315863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12686.320120] Lustre: Skipped 1 previous similar message [12686.330675] Lustre: Skipped 2 previous similar messages [12691.424989] LustreError: 419489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12691.447257] LustreError: 419489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12691.467858] Lustre: server umount lustre-MDT0000 complete [12694.603515] LustreError: 418455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198421 with bad export cookie 13106779892968171950 [12694.605471] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12694.614037] LustreError: 418455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12694.928909] Lustre: server umount lustre-MDT0001 complete [12704.385858] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12706.736823] Key type lgssc unregistered [12706.989926] LNet: 422016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12706.997672] LNetError: 422016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12707.014298] LNet: Removed LNI 192.168.204.145@tcp [12707.740254] Key type .llcrypt unregistered [12707.742372] Key type ._llcrypt unregistered [12729.539903] Key type ._llcrypt registered [12729.547602] Key type .llcrypt registered [12729.639613] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12730.514799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12730.571553] alg: No test for adler32 (adler32-zlib) [12731.556340] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [12731.742993] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12733.440710] Key type lgssc registered [12734.281661] Lustre: Echo OBD driver; http://www.lustre.org/ [12740.575424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12740.593323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12741.977144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12745.579308] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12751.330279] LustreError: 423432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12751.353771] LustreError: 423432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12753.441994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12753.746150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12757.177491] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12763.132527] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12768.073021] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12773.142446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12773.403337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12778.682523] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12784.851749] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12786.750809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12794.849313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12794.855637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12794.871128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12796.624704] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [12796.923546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12796.928175] Lustre: Skipped 1 previous similar message [12799.186737] Lustre: server umount lustre-MDT0000 complete [12801.036686] LustreError: 423413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1784198528 with bad export cookie 11564468095770923813 [12802.017877] LustreError: 423432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12802.029102] LustreError: 423432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12802.520567] LustreError: 423411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198529 with bad export cookie 11564468095770923148 [12802.528218] LustreError: 423411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12802.528975] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12804.066940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12804.072381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12804.090179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12804.094018] Lustre: Skipped 3 previous similar messages [12807.137349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12807.137705] LustreError: 423431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.160377] LustreError: 423431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12808.820701] Lustre: server umount lustre-OST0000 complete [12812.256340] LustreError: 423432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12812.269373] LustreError: 423432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12814.220913] Lustre: server umount lustre-MDT0001 complete [12821.936361] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12823.796692] Key type lgssc unregistered [12824.056954] LNet: 427116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12824.068921] LNetError: 427116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12824.083409] LNet: Removed LNI 192.168.204.145@tcp [12824.743168] Key type .llcrypt unregistered [12824.745761] Key type ._llcrypt unregistered [12844.195444] Key type ._llcrypt registered [12844.198909] Key type .llcrypt registered [12844.286525] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12844.920045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12844.968728] alg: No test for adler32 (adler32-zlib) [12846.009912] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [12846.194932] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12847.863250] Key type lgssc registered [12848.740745] Lustre: Echo OBD driver; http://www.lustre.org/ [12854.809122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12854.823480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12856.187044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12859.723740] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12866.490932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12866.528796] LustreError: 428513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.543712] LustreError: 428513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12866.775422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12871.335731] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12878.667303] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12885.805855] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12891.491114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12891.618036] LustreError: 430336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12891.876264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12897.206269] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [12901.364515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12902.789184] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12906.468183] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12906.478606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12906.483976] Lustre: Skipped 1 previous similar message [12909.782883] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [12911.585867] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12911.593383] Lustre: Skipped 1 previous similar message [12911.856765] Lustre: server umount lustre-MDT0001 complete [12916.706674] LustreError: 428513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12916.707377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12916.726056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.733540] Lustre: Skipped 1 previous similar message [12916.737345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12921.824828] LustreError: 428514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.827615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12922.262425] Lustre: server umount lustre-OST0000 complete [12924.978585] Lustre: server umount lustre-MDT0000 complete [12934.941696] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [12936.958560] Key type lgssc unregistered [12937.221744] LNet: 432196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12937.227588] LNetError: 432196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12937.244351] LNet: Removed LNI 192.168.204.145@tcp [12937.937552] Key type .llcrypt unregistered [12937.941754] Key type ._llcrypt unregistered [12956.402595] Key type ._llcrypt registered [12956.404815] Key type .llcrypt registered [12956.504549] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [12968.669589] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [12969.616798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12969.662370] alg: No test for adler32 (adler32-zlib) [12970.660542] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [12970.856323] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [12972.495215] Key type lgssc registered [12973.310608] Lustre: Echo OBD driver; http://www.lustre.org/ [13002.914914] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13014.616033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13014.632036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13015.841714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13015.865421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13015.920666] Lustre: lustre-MDT0000: new disk, initializing [13016.002409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13016.016705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13019.330782] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13030.572515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13030.758863] Lustre: lustre-OST0000: new disk, initializing [13030.765450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13030.768501] Lustre: Skipped 1 previous similar message [13030.772476] Lustre: 437207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13030.827313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13031.673407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13031.681705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13031.708135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13035.204930] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13042.518363] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13054.139354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13054.222349] Lustre: 436189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13054.286644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13054.352218] Lustre: lustre-MDT0001: new disk, initializing [13054.475989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13054.520632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13054.530092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13057.932460] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13065.158816] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13065.316499] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13068.628511] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13068.829544] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13073.652589] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [13077.474927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13077.478421] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13077.497256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13077.510427] Lustre: Skipped 1 previous similar message [13078.730904] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [13078.818953] Lustre: server umount lustre-MDT0001 complete [13087.719080] LustreError: 436195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.740128] LustreError: 436195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13092.844744] LustreError: 436198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13092.856172] LustreError: 436198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13097.955223] LustreError: 436199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13097.960152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13097.996551] Lustre: Skipped 1 previous similar message [13098.006921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13100.837591] Lustre: server umount lustre-OST0000 complete [13103.073713] LustreError: 436198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13109.806977] Lustre: server umount lustre-MDT0000 complete [13119.158897] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13120.889551] Key type lgssc unregistered [13121.113812] LNet: 440950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13121.120705] LNetError: 440950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13121.140410] LNet: Removed LNI 192.168.204.145@tcp [13121.688752] Key type .llcrypt unregistered [13121.690915] Key type ._llcrypt unregistered [13147.280278] Key type ._llcrypt registered [13147.283086] Key type .llcrypt registered [13147.374952] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13147.587789] Key type .llcrypt unregistered [13147.589944] Key type ._llcrypt unregistered [13158.121724] Key type ._llcrypt registered [13158.124561] Key type .llcrypt registered [13158.183834] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13158.843405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13158.857986] alg: No test for adler32 (adler32-zlib) [13159.779069] Lustre: Lustre: Build Version: 2.17.54_162_g9958c38 [13159.933698] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [13161.551162] Key type lgssc registered [13162.303658] Lustre: Echo OBD driver; http://www.lustre.org/ [13177.909824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13177.924622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13179.062471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13179.081044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13179.132644] Lustre: lustre-MDT0000: new disk, initializing [13179.179656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13179.188444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13181.671481] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13188.412415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13188.478443] Lustre: 444303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13188.519606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13188.523437] Lustre: Skipped 1 previous similar message [13188.568910] Lustre: lustre-MDT0001: new disk, initializing [13188.626644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13188.654617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13188.663122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13191.015414] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13196.814646] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13199.974918] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13203.666489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13203.817578] Lustre: lustre-OST0000: new disk, initializing [13203.819909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13203.824423] Lustre: 446269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13203.870509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13206.991426] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13209.119212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13209.129099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13209.214482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13212.576625] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13219.296274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13219.302225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13219.304271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13219.325164] Lustre: Skipped 2 previous similar messages [13222.539475] Lustre: server umount lustre-MDT0000 complete [13224.790484] LustreError: 444294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784198951 with bad export cookie 5755601725871407254 [13224.793478] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13224.803120] LustreError: 444294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13225.012354] Lustre: server umount lustre-MDT0001 complete [13237.586648] Lustre: server umount lustre-OST0000 complete [13244.193365] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:49:29 (1784198969) [13249.053479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13249.354413] LustreError: 448292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13249.369187] LustreError: 448292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13249.412706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13252.155710] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13254.630096] LustreError: 448293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13258.195201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13258.452045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13261.042177] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13265.758380] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13269.262579] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13286.186781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13286.309874] Lustre: Found index 0 for lustre-OST0000, updating log [13286.441849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13290.259078] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13294.637969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13295.073277] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13299.684178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13299.691489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.702733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13299.706173] Lustre: Skipped 2 previous similar messages [13300.768775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13303.794866] Lustre: server umount lustre-OST0000 complete [13317.663882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13317.666052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13317.675368] Lustre: Skipped 1 previous similar message [13323.862476] Lustre: server umount lustre-MDT0000 complete [13325.794031] LustreError: 448289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.802965] LustreError: 448289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13325.906411] LustreError: 450131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784199052 with bad export cookie 5755601725871408647 [13325.908195] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13325.914724] LustreError: 450131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13326.116157] Lustre: server umount lustre-MDT0001 complete [13330.877908] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:50:56 (1784199056) [13346.335331] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [13914.862058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13915.014143] LustreError: 454664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13915.041336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13916.440403] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13919.470956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13919.591690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13920.941723] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13923.365677] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13925.228226] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13940.515084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13940.620537] Lustre: Found index 0 for lustre-OST0000, updating log [13940.731893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13941.740504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13942.910594] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [13945.461624] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13946.847838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13946.851563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13946.858621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13946.861732] Lustre: Skipped 5 previous similar messages [13947.874614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13947.877794] Lustre: Skipped 1 previous similar message [13948.245782] Lustre: server umount lustre-OST0000 complete [13950.431821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13950.434269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13950.438207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13952.992125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13952.993687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13955.732372] Lustre: server umount lustre-MDT0000 complete [13956.064301] LustreError: 454664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13956.069884] LustreError: 454664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13956.953194] LustreError: 454645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784199683 with bad export cookie 5755601725871409718 [13956.955784] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13956.956419] LustreError: 454645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13957.070925] Lustre: server umount lustre-MDT0001 complete [13974.047291] LNet: 1 local NIs in recovery (showing 1): 192.168.204.145@tcp [13974.547100] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [13977.868733] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [13990.150952] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:01:56 (1784199716) [13996.898399] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [13998.039978] Key type lgssc unregistered [14002.271395] LNet: 461277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14002.275800] LNetError: 461277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found