[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495671243 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002387] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003019] kvm-guest: setup PV IPIs [ 0.006723] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009169] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012092] *** VALIDATE selinux *** [ 0.020000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024869] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027157] *** VALIDATE tmpfs *** [ 0.029332] *** VALIDATE proc *** [ 0.030305] *** VALIDATE cgroup *** [ 0.031017] *** VALIDATE cgroup2 *** [ 0.032310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033212] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035040] Spectre V2 : User space: Vulnerable [ 0.036015] Speculative Store Bypass: Vulnerable [ 0.039176] debug: unmapping init [mem 0xffffffff99059000-0xffffffff99060fff] [ 0.042217] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043826] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044033] ... version: 2 [ 0.045017] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049018] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.052221] rcu: Hierarchical SRCU implementation. [ 0.054697] smp: Bringing up secondary CPUs ... [ 0.055646] x86: Booting SMP configuration: [ 0.056035] .... node #0, CPUs: #1 #2 #3 [ 0.060451] smp: Brought up 1 node, 4 CPUs [ 0.062023] smpboot: Max logical packages: 1 [ 0.063029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.216022] node 0 deferred pages initialised in 151ms [ 0.219129] devtmpfs: initialized [ 0.220244] x86/mm: Memory block size: 128MB [ 0.222787] gcov: version magic: 0x41383552 [ 0.224229] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.225088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.227320] pinctrl core: initialized pinctrl subsystem [ 0.229163] [ 0.229703] ************************************************************* [ 0.231011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.233016] ** ** [ 0.235012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.238013] ** ** [ 0.240015] ** This means that this kernel is built to expose internal ** [ 0.242015] ** IOMMU data structures, which may compromise security on ** [ 0.244018] ** your system. ** [ 0.247017] ** ** [ 0.249013] ** If you see this message and you are not debugging the ** [ 0.251012] ** kernel, report this immediately to your vendor! ** [ 0.254013] ** ** [ 0.256015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258010] ************************************************************* [ 0.259650] NET: Registered protocol family 16 [ 0.261456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.264060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.266062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.270115] cpuidle: using governor menu [ 0.273816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275673] PCI: Using configuration type 1 for base access [ 0.278176] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.288126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.289040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.291073] cryptd: max_cpu_qlen set to 1000 [ 0.293657] ACPI: Added _OSI(Module Device) [ 0.295015] ACPI: Added _OSI(Processor Device) [ 0.296012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.301411] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.307236] ACPI: Interpreter enabled [ 0.308047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309008] ACPI: Using IOAPIC for interrupt routing [ 0.310097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323512] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.326052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335237] acpiphp: Slot [2] registered [ 0.337137] acpiphp: Slot [3] registered [ 0.339133] acpiphp: Slot [4] registered [ 0.340073] acpiphp: Slot [5] registered [ 0.342199] acpiphp: Slot [6] registered [ 0.343111] acpiphp: Slot [7] registered [ 0.345659] acpiphp: Slot [8] registered [ 0.347134] acpiphp: Slot [9] registered [ 0.348132] acpiphp: Slot [10] registered [ 0.350129] acpiphp: Slot [11] registered [ 0.352117] acpiphp: Slot [12] registered [ 0.354111] acpiphp: Slot [13] registered [ 0.355141] acpiphp: Slot [14] registered [ 0.357110] acpiphp: Slot [15] registered [ 0.358123] acpiphp: Slot [16] registered [ 0.360091] acpiphp: Slot [17] registered [ 0.361100] acpiphp: Slot [18] registered [ 0.363121] acpiphp: Slot [19] registered [ 0.364102] acpiphp: Slot [20] registered [ 0.366104] acpiphp: Slot [21] registered [ 0.367103] acpiphp: Slot [22] registered [ 0.369108] acpiphp: Slot [23] registered [ 0.370082] acpiphp: Slot [24] registered [ 0.371104] acpiphp: Slot [25] registered [ 0.373131] acpiphp: Slot [26] registered [ 0.374090] acpiphp: Slot [27] registered [ 0.376101] acpiphp: Slot [28] registered [ 0.377104] acpiphp: Slot [29] registered [ 0.379106] acpiphp: Slot [30] registered [ 0.380085] acpiphp: Slot [31] registered [ 0.381059] PCI host bridge to bus 0000:00 [ 0.383018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.391038] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.394027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.395178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399509] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.404514] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.414798] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.418978] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.421032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.424025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.427025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429502] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.432783] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.435052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.438429] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.443021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.456018] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.461902] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.466522] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.473018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.478019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.496019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.504417] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.512018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.518042] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536020] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.545909] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.550000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.568020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.574892] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.602018] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.611395] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.619017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.625014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642027] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.654794] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.661022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.668022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681036] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.693494] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.695419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.698387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.700406] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.707142] iommu: Default domain type: Passthrough [ 0.709409] SCSI subsystem initialized [ 0.710109] ACPI: bus type USB registered [ 0.712228] usbcore: registered new interface driver usbfs [ 0.714085] usbcore: registered new interface driver hub [ 0.716094] usbcore: registered new device driver usb [ 0.717150] pps_core: LinuxPPS API ver. 1 registered [ 0.719010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722114] PTP clock support registered [ 0.724120] EDAC MC: Ver: 3.0.0 [ 0.726212] PCI: Using ACPI for IRQ routing [ 0.728848] NetLabel: Initializing [ 0.730026] NetLabel: domain hash size = 128 [ 0.732013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.734090] NetLabel: unlabeled traffic allowed by default [ 0.736154] vgaarb: loaded [ 0.737250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.739015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745438] clocksource: Switched to clocksource kvm-clock [ 0.859878] VFS: Disk quotas dquot_6.6.0 [ 0.861747] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.864932] *** VALIDATE ramfs *** [ 0.866011] *** VALIDATE hugetlbfs *** [ 0.868130] pnp: PnP ACPI init [ 0.870197] pnp: PnP ACPI: found 6 devices [ 0.890722] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.893677] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895403] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.897782] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900052] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.902679] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.905710] NET: Registered protocol family 2 [ 0.908179] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.913159] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.916923] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.922146] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.925803] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928467] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.931373] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933607] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936329] NET: Registered protocol family 1 [ 0.939904] RPC: Registered named UNIX socket transport module. [ 0.941499] RPC: Registered udp transport module. [ 0.942634] RPC: Registered tcp transport module. [ 0.943899] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.946397] NET: Registered protocol family 44 [ 0.947977] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.950051] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.952084] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.954248] PCI: CLS 0 bytes, default 64 [ 0.956725] Unpacking initramfs... [ 2.357218] debug: unmapping init [mem 0xffff9efa3cc54000-0xffff9efa3ffbffff] [ 2.361841] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.365128] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.369176] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.881449] Initialise system trusted keyrings [ 2.882882] Key type blacklist registered [ 2.884484] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.893530] zbud: loaded [ 2.896377] *** VALIDATE nfs *** [ 2.897333] *** VALIDATE nfs4 *** [ 2.898844] pstore: using deflate compression [ 2.901920] Platform Keyring initialized [ 3.007021] NET: Registered protocol family 38 [ 3.009077] Key type asymmetric registered [ 3.010514] Asymmetric key parser 'x509' registered [ 3.012313] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.015634] io scheduler mq-deadline registered [ 3.017375] io scheduler kyber registered [ 3.019032] io scheduler bfq registered [ 3.020924] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.023889] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.026597] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.029582] ACPI: Power Button [PWRF] [ 3.123599] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214815] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.398268] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.491627] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.682719] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.713030] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.743768] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.749429] Non-volatile memory driver v1.3 [ 3.751081] Linux agpgart interface v0.103 [ 3.782226] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.786552] vda: detected capacity change from 0 to 68526080 [ 3.800738] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.803319] vdb: detected capacity change from 0 to 1073741824 [ 3.817729] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.821128] vdc: detected capacity change from 0 to 2621440000 [ 3.835636] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.838846] vdd: detected capacity change from 0 to 2621440000 [ 3.856106] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.859368] vde: detected capacity change from 0 to 4294967296 [ 3.877417] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.880227] vdf: detected capacity change from 0 to 4294967296 [ 3.892171] libphy: Fixed MDIO Bus: probed [ 3.901311] usbcore: registered new interface driver usbserial_generic [ 3.903952] usbserial: USB Serial support registered for generic [ 3.906634] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.911274] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.913293] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.916036] mousedev: PS/2 mouse device common for all mice [ 3.918797] rtc_cmos 00:05: RTC can wake from S4 [ 3.921930] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.925744] rtc_cmos 00:05: registered as rtc0 [ 3.927744] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.928638] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.933702] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.934535] intel_pstate: CPU model not supported [ 3.940941] hid: raw HID events driver (C) Jiri Kosina [ 3.943329] usbcore: registered new interface driver usbhid [ 3.945274] usbhid: USB HID core driver [ 3.947283] drop_monitor: Initializing network drop monitor service [ 3.950144] Initializing XFRM netlink socket [ 3.952320] NET: Registered protocol family 10 [ 3.955344] Segment Routing with IPv6 [ 3.956816] NET: Registered protocol family 17 [ 3.958902] mpls_gso: MPLS GSO support [ 3.965612] RAS: Correctable Errors collector initialized. [ 3.969119] AVX version of gcm_enc/dec engaged. [ 3.970659] AES CTR mode by8 optimization enabled [ 4.057183] sched_clock: Marking stable (4057162890, 0)->(4985288342, -928125452) [ 4.060980] registered taskstats version 1 [ 4.065267] Loading compiled-in X.509 certificates [ 4.067363] zswap: loaded using pool lzo/zbud [ 4.092906] Key type big_key registered [ 4.104453] Key type encrypted registered [ 4.105828] ima: No TPM chip found, activating TPM-bypass! [ 4.108035] ima: Allocated hash algorithm: sha1 [ 4.109777] ima: No architecture policies found [ 4.111590] evm: Initialising EVM extended attributes: [ 4.113465] evm: security.selinux [ 4.114655] evm: security.ima [ 4.115806] evm: security.capability [ 4.117080] evm: HMAC attrs: 0x1 [ 4.119456] rtc_cmos 00:05: setting system clock to 2025-11-17 01:09:54 UTC (1763341794) [ 4.124794] debug: unmapping init [mem 0xffffffff9a003000-0xffffffff9a1fffff] [ 4.127726] debug: unmapping init [mem 0xffffffff98d82000-0xffffffff99058fff] [ 4.136192] Write protecting the kernel read-only data: 28672k [ 4.139036] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 4.141941] debug: unmapping init [mem 0xffffffff97d14000-0xffffffff97dfffff] [ 4.176069] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.184329] systemd[1]: Detected virtualization kvm. [ 4.186265] systemd[1]: Detected architecture x86-64. [ 4.187716] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.212417] systemd[1]: No hostname configured. [ 4.214293] systemd[1]: Set hostname to . [ 4.217372] random: systemd: uninitialized urandom read (16 bytes read) [ 4.219332] systemd[1]: Initializing machine ID from random generator. [ 4.280745] random: ln: uninitialized urandom read (6 bytes read) [ 4.364879] random: systemd: uninitialized urandom read (16 bytes read) [ 4.367231] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.371969] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.376519] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.979271] device-mapper: uevent: version 1.0.3 [ 4.981613] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.641063] virtio_net virtio0 ens2: renamed from eth0 [ 5.645690] random: fast init done [ 5.678315] scsi host0: ata_piix [ 5.689691] scsi host1: ata_piix [ 5.691393] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.694261] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.290310] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.621321] random: crng init done [ 10.622606] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.896067] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.051599] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.302278] SELinux: Disabled at runtime. [ 12.360776] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.369811] systemd[1]: Detected virtualization kvm. [ 12.372169] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.880120] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.883596] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.888762] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.892374] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.895378] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.901941] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.910446] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 12.957442] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.440782] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.783496] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.815118] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.902080] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.918177] EDAC sbridge: Ver: 1.1.2 [ 15.604680] Key type dns_resolver registered [ 15.904588] NFS: Registering the id_resolver key type [ 15.907535] Key type id_resolver registered [ 15.909533] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-server login: [ 62.735305] libcfs: loading out-of-tree module taints kernel. [ 62.789439] Key type ._llcrypt registered [ 62.791213] Key type .llcrypt registered [ 62.904980] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 78.568632] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 79.719600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.744681] alg: No test for adler32 (adler32-zlib) [ 81.100932] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 81.804206] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 83.514672] Key type lgssc registered [ 84.956253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.370852] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 100.246614] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.213522] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.482032] hrtimer: interrupt took 7977888 ns [ 113.040936] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.795563] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.670199] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 145.791822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.860913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.872271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.197678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.229906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.310733] Lustre: lustre-MDT0000: new disk, initializing [ 147.427097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.452507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.314411] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.140192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.213938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 164.296232] Lustre: 6510:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.329903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.333078] Lustre: Skipped 1 previous similar message [ 164.448769] Lustre: lustre-MDT0001: new disk, initializing [ 164.548293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.589525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.598953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.383477] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.312911] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.255072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.352941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 184.524766] Lustre: lustre-OST0000: new disk, initializing [ 184.527379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.579533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.433600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.446825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.563813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 191.380491] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.256036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.317808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.402613] Lustre: lustre-OST0001: new disk, initializing [ 205.417236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.472743] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.468402] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.515291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.528511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.621633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.250942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.908048] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.899708] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.411190] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing yml_node [ 245.932343] Lustre: DEBUG MARKER: Client: 2.16.61.48 [ 248.306356] Lustre: DEBUG MARKER: MDS: 2.16.61.48 [ 250.703787] Lustre: DEBUG MARKER: OSS: 2.16.61.48 [ 252.263474] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 20:14:00 EST 2025 [ 269.034808] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 270.551584] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.713855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.718670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.733240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.226321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.253423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 288.256493] Lustre: Skipped 3 previous similar messages [ 291.613055] LustreError: 12218:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 293.673140] LustreError: 12218:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 293.682029] LustreError: 12218:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 293.753930] Lustre: server umount lustre-MDT0000 complete [ 297.277434] LustreError: 7896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342087 with bad export cookie 10537771291207710962 [ 297.282885] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.299280] LustreError: 7896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 298.465784] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.466688] LustreError: 7795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.466860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.480326] Lustre: Skipped 3 previous similar messages [ 298.523962] LustreError: 7795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 302.496871] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.496957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.523514] Lustre: Skipped 2 previous similar messages [ 303.554401] LustreError: 12420:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 303.560970] LustreError: 12420:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 303.783679] Lustre: server umount lustre-MDT0001 complete [ 307.425121] LustreError: 12621:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 307.427843] LustreError: 12621:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 307.511658] Lustre: server umount lustre-OST0000 complete [ 311.176244] Lustre: server umount lustre-OST0001 complete [ 316.232860] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 324.861473] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 334.320770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.527355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.724603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.088954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 359.106911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 359.158555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.383838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 359.426986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 359.508487] Lustre: lustre-MDT0000: new disk, initializing [ 359.645224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 359.656651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 363.097076] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 374.359482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 374.430165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 374.518635] Lustre: 15009:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.595203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.597852] Lustre: Skipped 1 previous similar message [ 374.706835] Lustre: lustre-MDT0001: new disk, initializing [ 374.839703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.904275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.935211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.631392] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 387.049485] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 391.119570] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 397.826513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 397.914034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 398.219098] Lustre: lustre-OST0000: new disk, initializing [ 398.226517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 398.320332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.559353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 399.565517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 399.658681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.091782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 411.307906] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 415.917259] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 416.165514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 419.714290] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 419.924529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 425.443827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 425.445880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.460270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 425.463679] Lustre: Skipped 1 previous similar message [ 435.689277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.696593] Lustre: Skipped 4 previous similar messages [ 436.192350] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 436.227096] LustreError: 17947:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 436.234296] LustreError: 17947:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 436.345884] Lustre: server umount lustre-OST0000 complete [ 440.804072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.810805] Lustre: Skipped 1 previous similar message [ 454.112170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 454.221109] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 454.228141] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 454.286384] Lustre: server umount lustre-MDT0000 complete [ 456.163792] LustreError: 16110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.176995] LustreError: 16110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 457.450264] LustreError: 15000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342247 with bad export cookie 10537771291207712502 [ 457.452396] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.462848] LustreError: 15000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 457.752850] Lustre: server umount lustre-MDT0001 complete [ 461.718670] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:17:29 (1763342249) [ 463.806392] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 465.496131] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:17:33 (1763342253) [ 471.989501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.435516] LustreError: 19021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.497749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 475.809393] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 477.669835] LustreError: 19020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.901729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 488.170187] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 493.213938] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.563826] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.643967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 509.777213] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 516.015816] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 530.922781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 530.935443] LustreError: Skipped 1 previous similar message [ 530.937316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.960106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 530.977150] Lustre: Skipped 6 previous similar messages [ 532.976754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 535.250143] LustreError: 21291:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 535.253216] LustreError: 21291:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 535.551101] Lustre: server umount lustre-OST0000 complete [ 540.641534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 546.377726] Lustre: server umount lustre-MDT0000 complete [ 548.321456] LustreError: 19020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.338943] LustreError: 19020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 550.525124] LustreError: 19002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342340 with bad export cookie 10537771291207713384 [ 550.528771] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.547455] LustreError: 19002:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 550.798114] Lustre: server umount lustre-MDT0001 complete [ 560.103857] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 562.537743] Key type lgssc unregistered [ 562.829505] LNet: 22226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.835685] LNetError: 22226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.847573] LNet: Removed LNI 192.168.202.153@tcp [ 563.541164] Key type .llcrypt unregistered [ 563.543184] Key type ._llcrypt unregistered [ 581.759972] Key type ._llcrypt registered [ 581.761710] Key type .llcrypt registered [ 581.866299] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 582.807514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 582.836064] alg: No test for adler32 (adler32-zlib) [ 583.879357] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 584.141539] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 585.832290] Key type lgssc registered [ 586.888341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 593.386914] loop: module loaded [ 593.441534] loop0: detected capacity change from 0 to 204800000 [ 593.568650] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.578428] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.812776] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.858792] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 598.904358] loop0: detected capacity change from 0 to 204800000 [ 599.081023] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.093674] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.116156] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.250816] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 604.031337] loop0: detected capacity change from 0 to 204800000 [ 610.314809] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 610.368743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 610.390346] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.623089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 611.654702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 611.725424] Lustre: lustre-MDT0000: new disk, initializing [ 611.811050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.842600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 615.166423] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 624.347403] loop1: detected capacity change from 0 to 204800000 [ 630.708949] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 630.765761] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 631.161593] Lustre: lustre-OST07c6: new disk, initializing [ 631.164133] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 631.167076] Lustre: Skipped 1 previous similar message [ 631.379433] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 638.371713] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 640.163322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 640.177183] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 640.215469] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 644.755372] Lustre: 24369:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 644.793424] Lustre: 24369:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 644.817205] Lustre: 24369:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 644.845685] Lustre: 24369:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 644.867698] Lustre: 24369:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 644.893807] Lustre: 24369:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 662.478658] Lustre: 24368:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 662.494482] Lustre: 24368:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 662.500744] Lustre: 24368:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 662.519040] Lustre: 24368:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 662.523314] Lustre: 24368:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 662.526168] Lustre: 24368:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 667.149913] Lustre: 24368:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 667.165573] Lustre: 24368:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 667.171593] Lustre: 24368:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 667.192945] Lustre: 24368:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.202151] Lustre: 24368:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 667.217526] Lustre: 24368:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.229067] Lustre: 24368:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 667.239308] Lustre: 24368:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.244477] Lustre: 24368:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 667.253984] Lustre: 24368:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.262042] Lustre: 24368:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 667.271486] Lustre: 24368:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.486314] Lustre: 24370:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 670.499837] Lustre: 24370:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 670.518733] Lustre: 24370:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 670.533819] Lustre: 24370:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.537481] Lustre: 24370:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 670.563182] Lustre: 24370:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.570630] Lustre: 24370:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 670.577751] Lustre: 24370:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.582390] Lustre: 24370:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 670.590020] Lustre: 24370:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.601732] Lustre: 24370:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.613418] Lustre: 24370:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.428974] Lustre: 26039:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 675.442487] Lustre: 26039:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 675.465176] Lustre: 26039:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 675.467861] Lustre: 26039:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.486035] Lustre: 26039:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 675.495393] Lustre: 26039:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.509699] Lustre: 26039:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 675.524158] Lustre: 26039:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.536487] Lustre: 26039:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 675.552223] Lustre: 26039:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.572407] Lustre: 26039:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 675.588984] Lustre: 26039:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.225334] Lustre: 26039:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 686.231252] Lustre: 26039:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 686.236596] Lustre: 26039:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 686.241866] Lustre: 26039:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.246642] Lustre: 26039:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.252379] Lustre: 26039:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.263646] Lustre: 26039:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 686.271093] Lustre: 26039:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.282671] Lustre: 26039:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.297280] Lustre: 26039:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.312401] Lustre: 26039:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.330226] Lustre: 26039:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 737.254316] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.273223] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 740.302395] LustreError: 26557:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 740.413848] Lustre: server umount lustre-OST07c6 complete [ 749.379652] LustreError: 27060:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 749.388419] LustreError: 27060:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 749.614841] Lustre: server umount lustre-MDT0000 complete [ 766.353042] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 769.230088] Key type lgssc unregistered [ 769.514708] LNet: 27894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.533059] LNetError: 27894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.603778] LNet: Removed LNI 192.168.202.153@tcp [ 771.467154] Key type .llcrypt unregistered [ 771.468918] Key type ._llcrypt unregistered [ 788.981487] Key type ._llcrypt registered [ 788.982997] Key type .llcrypt registered [ 789.098086] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 789.910265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.943818] alg: No test for adler32 (adler32-zlib) [ 791.050900] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 791.244335] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 792.896168] Key type lgssc registered [ 794.057362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.350037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.383648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.890630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.366703] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 813.038742] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.068507] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 814.721195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 815.131880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 819.105738] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 824.968690] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 828.673231] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 835.520245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 835.944068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.024425] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 846.267923] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 848.254784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 868.037111] LustreError: 31599:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 868.158836] Lustre: server umount lustre-OST0000 complete [ 873.953631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.959309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.966617] Lustre: Skipped 1 previous similar message [ 873.986381] Lustre: Skipped 1 previous similar message [ 879.074859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.084192] Lustre: Skipped 1 previous similar message [ 882.176691] LustreError: 31800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 882.179546] LustreError: 31800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 882.276553] Lustre: server umount lustre-MDT0000 complete [ 884.193362] LustreError: 29143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.208768] LustreError: 29143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 885.216556] Lustre: 28456:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342659/real 1763342659] req@000000000d14cebf x1848998710244992/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763342675 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 885.260323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.276442] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.303884] LustreError: 29148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 886.023588] LustreError: 29126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342676 with bad export cookie 9343329131853862086 [ 886.029523] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 886.231559] LustreError: 32002:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 886.246252] LustreError: 32002:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 886.595756] Lustre: server umount lustre-MDT0001 complete [ 896.283814] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 898.851530] Key type lgssc unregistered [ 899.218620] LNet: 32534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.228531] LNetError: 32534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.242035] LNet: Removed LNI 192.168.202.153@tcp [ 899.997504] Key type .llcrypt unregistered [ 900.001415] Key type ._llcrypt unregistered [ 918.467225] Key type ._llcrypt registered [ 918.469548] Key type .llcrypt registered [ 918.580113] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 919.925832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 920.002089] alg: No test for adler32 (adler32-zlib) [ 921.129737] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 921.434141] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 923.177070] Key type lgssc registered [ 924.330714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 933.122544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 933.158646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.876730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.172233] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 945.127564] LustreError: 33787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.142531] LustreError: 33787:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 947.733792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.300769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.704183] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 957.943335] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 961.848757] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 968.305412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 968.602340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.229671] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 979.889939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 979.989749] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 990.177333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 990.190174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.205476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.244098] Lustre: Skipped 1 previous similar message [ 995.301189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 995.306270] Lustre: Skipped 2 previous similar messages [ 995.601097] LustreError: 36035:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 995.753352] Lustre: server umount lustre-OST0000 complete [ 1000.418714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.423818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.541380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.551932] Lustre: Skipped 2 previous similar messages [ 1010.664973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.667753] Lustre: Skipped 1 previous similar message [ 1014.240118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.363262] LustreError: 36236:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1014.365640] LustreError: 36236:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1014.491256] Lustre: server umount lustre-MDT0000 complete [ 1015.778126] LustreError: 33788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.793947] LustreError: 33788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1018.357414] LustreError: 33769:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342808 with bad export cookie 2790394409077559415 [ 1018.366288] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.378432] LustreError: 33769:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1018.534918] LustreError: 36439:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1018.546768] LustreError: 36439:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1018.700443] Lustre: server umount lustre-MDT0001 complete [ 1029.865775] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1033.203836] Key type lgssc unregistered [ 1033.446864] LNet: 36972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.484913] LNetError: 36972:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.511199] LNet: Removed LNI 192.168.202.153@tcp [ 1034.439380] Key type .llcrypt unregistered [ 1034.441479] Key type ._llcrypt unregistered [ 1057.947667] Key type ._llcrypt registered [ 1057.957749] Key type .llcrypt registered [ 1058.059490] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1059.401138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.476523] alg: No test for adler32 (adler32-zlib) [ 1060.542713] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1060.780100] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1062.496171] Key type lgssc registered [ 1063.603189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.126665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.158805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.617088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.534538] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1081.831770] LustreError: 38370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.842161] LustreError: 38370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1084.493956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1084.849058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1088.960735] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1095.083854] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1099.851791] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1107.397721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1107.735270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.621528] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1116.978528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1119.173336] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.186672] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 1127.394030] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.409800] Lustre: Skipped 1 previous similar message [ 1132.270807] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 1132.286364] Lustre: Skipped 3 previous similar messages [ 1132.293325] LustreError: 40618:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1132.532850] LustreError: 39083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.846101] Lustre: server umount lustre-MDT0000 complete [ 1137.046215] LustreError: 38350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342927 with bad export cookie 3516697225249930419 [ 1137.055029] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.063271] LustreError: 38350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1137.382487] LustreError: 38366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.402750] LustreError: 38366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1137.447383] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [ 1137.642338] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.660916] Lustre: Skipped 1 previous similar message [ 1142.502524] LustreError: 38366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.506254] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [ 1142.519888] LustreError: 38366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1142.551257] Lustre: Skipped 1 previous similar message [ 1143.388244] LustreError: 40821:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1143.458031] LustreError: 40821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1143.755561] Lustre: server umount lustre-MDT0001 complete [ 1165.386833] LustreError: 41053:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1165.406610] LustreError: 41053:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1165.531972] Lustre: server umount lustre-OST0000 complete [ 1180.924353] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1183.710780] Key type lgssc unregistered [ 1183.968207] LNet: 41891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.983706] LNetError: 41891:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.999390] LNet: Removed LNI 192.168.202.153@tcp [ 1185.763880] Key type .llcrypt unregistered [ 1185.765523] Key type ._llcrypt unregistered [ 1210.175759] Key type ._llcrypt registered [ 1210.178380] Key type .llcrypt registered [ 1210.274399] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1211.087638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.247902] alg: No test for adler32 (adler32-zlib) [ 1212.323614] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1212.666682] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1214.400148] Key type lgssc registered [ 1215.515274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1222.694836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1222.711317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.110731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.438078] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1234.405308] LustreError: 43301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.426231] LustreError: 43301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1237.791212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1238.127254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.746851] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1248.003079] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1251.644816] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1259.543276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1259.922267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.335386] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1269.144695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1272.335549] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1278.436233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1278.442262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.460169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1279.257511] Lustre: lustre-OST0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 1279.460502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.463377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1283.792231] LustreError: 45547:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1284.014787] Lustre: server umount lustre-OST0000 complete [ 1303.714052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.740337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.753230] Lustre: Skipped 3 previous similar messages [ 1307.826282] LustreError: 45924:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1307.831556] LustreError: 45924:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1307.912838] Lustre: server umount lustre-MDT0000 complete [ 1309.171353] LustreError: 43296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1309.186358] LustreError: 43296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1311.312751] LustreError: 43283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343101 with bad export cookie 11423492088170827545 [ 1311.313372] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.321206] LustreError: 43283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1317.563450] LustreError: 46127:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1317.569539] LustreError: 46127:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1317.793214] Lustre: server umount lustre-MDT0001 complete [ 1327.926244] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1330.682697] Key type lgssc unregistered [ 1331.017688] LNet: 46659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.022144] LNetError: 46659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.045070] LNet: Removed LNI 192.168.202.153@tcp [ 1331.736110] Key type .llcrypt unregistered [ 1331.737691] Key type ._llcrypt unregistered [ 1352.388635] Key type ._llcrypt registered [ 1352.393495] Key type .llcrypt registered [ 1352.502238] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1353.596178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1353.657091] alg: No test for adler32 (adler32-zlib) [ 1354.702043] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1354.960827] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1356.698359] Key type lgssc registered [ 1357.654058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1365.180691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1365.214604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.681884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.584511] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1376.230146] LustreError: 48037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.257332] LustreError: 48037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1378.562142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1378.989123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1382.416606] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1388.085112] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1391.490917] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1398.398238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1398.789861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.925512] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1409.201521] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1411.077692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1418.408760] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1421.599024] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1433.883902] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1620h33m40s [ 1435.561826] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1440.225831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1440.229148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.254370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.315688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.325776] Lustre: Skipped 6 previous similar messages [ 1443.640141] LustreError: 50603:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1443.696538] Lustre: server umount lustre-MDT0000 complete [ 1446.688440] LustreError: 48030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.444226] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1452.772322] LustreError: 48032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.809935] LustreError: 48032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1454.502352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.642269] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1454.961659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.040087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1455.974636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1459.168751] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1462.885807] Lustre: DEBUG MARKER: Wait for df (36041) ... [ 1464.977985] Lustre: DEBUG MARKER: done [ 1470.449117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1470.450130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.474736] LustreError: Skipped 1 previous similar message [ 1470.479834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.496062] Lustre: Skipped 3 previous similar messages [ 1470.523078] Lustre: Skipped 2 previous similar messages [ 1474.469673] LustreError: 51661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1474.473323] LustreError: 51661:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1474.610598] Lustre: server umount lustre-OST0000 complete [ 1481.697151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.712900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.731859] Lustre: Skipped 3 previous similar messages [ 1484.640399] LustreError: 51863:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1484.644850] LustreError: 51863:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1484.727672] Lustre: server umount lustre-MDT0000 complete [ 1488.382438] LustreError: 48016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343278 with bad export cookie 9805471603092290200 [ 1488.398174] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.511761] LustreError: 52064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1488.514479] LustreError: 52064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1488.820127] Lustre: server umount lustre-MDT0001 complete [ 1497.940121] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1500.316391] Key type lgssc unregistered [ 1500.553587] LNet: 52596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.561920] LNetError: 52596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.574915] LNet: Removed LNI 192.168.202.153@tcp [ 1501.281810] Key type .llcrypt unregistered [ 1501.285423] Key type ._llcrypt unregistered [ 1532.257331] Key type ._llcrypt registered [ 1532.258942] Key type .llcrypt registered [ 1532.361777] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1546.154445] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1547.136060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.315441] alg: No test for adler32 (adler32-zlib) [ 1548.410223] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1548.617940] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1550.232179] Key type lgssc registered [ 1551.203690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1559.701848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1566.097832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1571.837593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1577.733178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1585.172896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1585.244789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1585.259441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1586.615773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1586.666891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1586.743579] Lustre: lustre-MDT0000: new disk, initializing [ 1586.844699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.867519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1591.354144] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1601.639982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1601.709808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1601.786034] Lustre: 56549:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1601.849845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1601.856975] Lustre: Skipped 1 previous similar message [ 1601.927396] Lustre: lustre-MDT0001: new disk, initializing [ 1602.100892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1602.175324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1602.188504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1605.895097] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1612.160912] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1615.646475] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1621.941872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1622.020404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1622.319475] Lustre: lustre-OST0000: new disk, initializing [ 1622.327186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1622.421631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.014885] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1628.206486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1628.219256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1628.299605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1633.157733] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1637.048140] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1637.277143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1641.163100] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1641.373864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1643.492127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.498913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1643.503904] Lustre: Skipped 1 previous similar message [ 1648.611117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1648.616235] Lustre: Skipped 1 previous similar message [ 1653.737293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1653.742342] Lustre: Skipped 1 previous similar message [ 1657.312140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1657.350156] LustreError: 59487:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1657.497386] Lustre: server umount lustre-OST0000 complete [ 1663.461489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1663.470866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.477861] Lustre: Skipped 1 previous similar message [ 1663.481703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.303767] LustreError: 59689:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1667.316434] LustreError: 59689:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1667.402711] Lustre: server umount lustre-MDT0000 complete [ 1669.089398] LustreError: 56556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.099429] LustreError: 56556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1670.392149] LustreError: 56542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343460 with bad export cookie 16128400785132475343 [ 1670.393593] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.402302] LustreError: 56542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1670.556600] LustreError: 59892:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1670.560314] LustreError: 59892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1670.816790] Lustre: server umount lustre-MDT0001 complete [ 1678.307795] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 20:37:46 (1763343466) [ 1684.632707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.990656] LustreError: 60563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.058940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.125937] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1690.085970] LustreError: 60564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.105905] LustreError: 60564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1695.203481] LustreError: 60564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.326224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.675260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.406834] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1704.113613] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1707.131629] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1712.106975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1712.348329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.270463] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1721.727810] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1736.597208] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1620h38m44s [ 1737.917180] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1742.815087] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1744.887896] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1744.948511] LustreError: 60560:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1765.167281] Lustre: lustre-MDT0000: Client 1bbe5aa6-9085-47a1-a73d-77868ba2e9b1 (at 192.168.202.53@tcp) reconnecting [ 1785.016202] LustreError: 60560:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1787.187736] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1789.958773] Lustre: DEBUG MARKER: done [ 1796.578641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1796.582082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.606160] Lustre: Skipped 1 previous similar message [ 1796.610259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.619563] Lustre: Skipped 2 previous similar messages [ 1802.189272] LustreError: 63590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1802.197992] LustreError: 63590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1802.314768] Lustre: server umount lustre-OST0000 complete [ 1806.822577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.823322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.830802] Lustre: Skipped 2 previous similar messages [ 1806.839094] Lustre: Skipped 3 previous similar messages [ 1820.129193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1820.256406] LustreError: 63791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1820.272797] LustreError: 63791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1820.330832] Lustre: server umount lustre-MDT0000 complete [ 1823.609405] LustreError: 60545:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343613 with bad export cookie 16128400785132476225 [ 1823.617174] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1823.643177] LustreError: 60545:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1823.925594] Lustre: server umount lustre-MDT0001 complete [ 1833.533603] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1836.405386] Key type lgssc unregistered [ 1836.667667] LNet: 64525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.683275] LNetError: 64525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.707896] LNet: Removed LNI 192.168.202.153@tcp [ 1837.696212] Key type .llcrypt unregistered [ 1837.697692] Key type ._llcrypt unregistered [ 1869.959452] Key type ._llcrypt registered [ 1869.980970] Key type .llcrypt registered [ 1870.164212] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1885.340868] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1886.455618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.499603] alg: No test for adler32 (adler32-zlib) [ 1887.542776] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1887.865902] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1889.552190] Key type lgssc registered [ 1890.686694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1898.522880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1904.501538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.708728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1915.535441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1921.530093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1921.595921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.615355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.781989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1922.804495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1922.866880] Lustre: lustre-MDT0000: new disk, initializing [ 1922.935480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.949258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1925.958843] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1934.566528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1934.631604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1934.703229] Lustre: 68477:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1934.763875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1934.766353] Lustre: Skipped 1 previous similar message [ 1934.836400] Lustre: lustre-MDT0001: new disk, initializing [ 1934.951771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.009210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1935.027565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1938.585702] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1946.180533] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.400780] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1956.028517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.118535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1956.522836] Lustre: lustre-OST0000: new disk, initializing [ 1956.525836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1956.631038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.799381] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1966.654792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1966.679821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1966.805127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1969.951932] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1974.497164] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1974.793497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1979.626975] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1979.945636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1982.435247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.439934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1982.457886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.565174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.579935] Lustre: Skipped 2 previous similar messages [ 1988.293210] LustreError: 71416:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1988.491729] Lustre: server umount lustre-OST0000 complete [ 1992.680512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.687734] Lustre: Skipped 1 previous similar message [ 1992.691504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.801783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.806739] Lustre: Skipped 2 previous similar messages [ 1998.381406] LustreError: 71618:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1998.384723] LustreError: 71618:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1998.452637] Lustre: server umount lustre-MDT0000 complete [ 2001.784835] LustreError: 69257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343792 with bad export cookie 14908585733776352695 [ 2001.789078] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.790336] LustreError: 69257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2001.933350] LustreError: 71820:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2001.935613] LustreError: 71820:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2002.071292] Lustre: server umount lustre-MDT0001 complete [ 2011.295413] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 20:43:18 (1763343798) [ 2013.463907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 2015.504740] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 20:43:23 (1763343803) [ 2028.610386] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2032.868867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2033.226155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.300508] Lustre: Failing over lustre-MDT0000 [ 2034.310318] LustreError: 72897:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2034.319220] LustreError: 72897:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2034.332052] LustreError: 72897:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 2034.522970] LustreError: 72948:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2034.533252] LustreError: 72948:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2034.802974] Lustre: server umount lustre-MDT0000 complete [ 2042.365698] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:43:49 (1763343829) [ 2049.107112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.635335] LustreError: 73517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.660255] LustreError: 73517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2049.706376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.529629] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2055.154187] LustreError: 73517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.259486] LustreError: 73517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.464569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2062.859888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2066.820070] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2072.793854] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2077.002673] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2085.074194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2085.514263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.529931] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2097.448885] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2103.159135] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2110.435906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.439616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.469594] Lustre: Skipped 2 previous similar messages [ 2115.468815] LustreError: 75861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2115.476479] LustreError: 75861:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2115.555094] LustreError: 73511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.581273] LustreError: 73511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2115.657879] Lustre: server umount lustre-MDT0000 complete [ 2119.411748] LustreError: 75863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343909 with bad export cookie 14908585733776353850 [ 2119.412565] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.421526] LustreError: 75863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2119.793628] Lustre: server umount lustre-MDT0001 complete [ 2123.143687] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2128.053903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2136.188481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.058729] Lustre: 66691:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343911/real 1763343911] req@000000005cb9c03b x1848999860573568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763343927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2137.098159] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.140733] Lustre: Skipped 1 previous similar message [ 2141.216155] Lustre: 66691:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343915/real 1763343915] req@0000000097c66211 x1848999860573696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763343931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.634342] LustreError: 75124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.728647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.808924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2145.707214] Lustre: 66690:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343919/real 1763343919] req@000000004351913b x1848999860574080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763343935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2145.748350] Lustre: 66690:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2149.196785] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2149.807324] Lustre: 66691:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343924/real 1763343924] req@000000006ed66670 x1848999860574208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763343940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2152.930344] LustreError: 76921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.977263] LustreError: 76921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2158.726037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2163.927159] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2164.711836] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2165.736348] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2170.348403] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2174.714920] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2181.964715] LustreError: 78328:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2182.020135] LustreError: 78328:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2182.138084] Lustre: server umount lustre-MDT0000 complete [ 2185.187658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2185.198039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.219239] LustreError: 75125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.237605] LustreError: 75125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2186.219127] LustreError: 73498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343976 with bad export cookie 14908585733776355754 [ 2186.237617] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.586384] Lustre: server umount lustre-MDT0001 complete [ 2192.560022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2202.230691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.693027] Lustre: 66692:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343981/real 1763343981] req@000000007901ebe1 x1848999860600576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763343997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.721788] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.728187] Lustre: Skipped 1 previous similar message [ 2211.110996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2211.113791] Lustre: Skipped 1 previous similar message [ 2211.165492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2214.409137] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2217.250749] Lustre: 66690:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343991/real 1763343991] req@0000000035af8095 x1848999860600960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763344007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2217.287253] Lustre: 66690:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2217.457841] LustreError: 75125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.473546] LustreError: 75125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2222.486392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2225.911552] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2226.312526] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2232.255959] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2235.921891] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2241.508458] LustreError: 66691:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f2eea06c x1848999860620672/t0(0) o41->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2241.509657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.516746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.528970] Lustre: Skipped 1 previous similar message [ 2241.540915] Lustre: Skipped 4 previous similar messages [ 2243.729851] LustreError: 80786:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2243.732273] LustreError: 80786:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2243.859252] Lustre: server umount lustre-MDT0000 complete [ 2247.294979] LustreError: 73498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344037 with bad export cookie 14908585733776357126 [ 2247.296170] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.304625] LustreError: 73498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2247.570656] Lustre: server umount lustre-MDT0001 complete [ 2253.365883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2256.387044] Lustre: DEBUG MARKER: files compared the same [ 2268.556322] Lustre: server umount lustre-OST0000 complete [ 2284.132017] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2286.511757] Key type lgssc unregistered [ 2286.838147] LNet: 82420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.848316] LNetError: 82420:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.861895] LNet: Removed LNI 192.168.202.153@tcp [ 2287.630639] Key type .llcrypt unregistered [ 2287.637416] Key type ._llcrypt unregistered [ 2310.285708] Key type ._llcrypt registered [ 2310.287432] Key type .llcrypt registered [ 2310.386910] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2311.296185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.313695] alg: No test for adler32 (adler32-zlib) [ 2312.652636] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2312.978243] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2314.715161] Key type lgssc registered [ 2316.082980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.685131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.717605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.257331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.867534] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug +malloc all [ 2334.690409] LustreError: 83818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.717944] LustreError: 83818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2336.861437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2337.084193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2340.548896] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug +malloc all [ 2346.799483] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2350.871559] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2357.572753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2357.879827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.551569] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug +malloc all [ 2368.817952] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2370.162888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2382.816841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.832458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.844386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2385.890542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.894085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.944386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.410467] LustreError: 86088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2387.592563] Lustre: server umount lustre-OST0000 complete [ 2393.569926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.577750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.588349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.598929] Lustre: Skipped 1 previous similar message [ 2396.132292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.132389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.304397] LustreError: 86289:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2397.318902] LustreError: 86289:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2397.479554] Lustre: server umount lustre-MDT0000 complete [ 2401.101360] LustreError: 83799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344191 with bad export cookie 13141635157476970877 [ 2401.104697] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2401.118880] LustreError: 83799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2401.316376] LustreError: 86493:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2401.319715] LustreError: 86493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2401.645559] Lustre: server umount lustre-MDT0001 complete [ 2412.676917] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2415.227952] Key type lgssc unregistered [ 2415.549109] LNet: 87024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.552767] LNetError: 87024:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.624730] LNet: Removed LNI 192.168.202.153@tcp [ 2417.381412] Key type .llcrypt unregistered [ 2417.382886] Key type ._llcrypt unregistered [ 2443.327768] Key type ._llcrypt registered [ 2443.333578] Key type .llcrypt registered [ 2443.410235] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2444.327351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.432762] alg: No test for adler32 (adler32-zlib) [ 2445.557876] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2445.824266] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2447.552960] Key type lgssc registered [ 2448.831767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2456.634053] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2456.667384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2472.224387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2477.344964] LustreError: 88393:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 2503.968269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2508.877408] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2515.475458] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2524.755122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.961450] Lustre: *** cfs_fail_loc=706, val=0*** [ 2525.578277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.670206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2530.790738] LustreError: 88418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.829731] LustreError: 88418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2530.847816] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2535.911786] LustreError: 88418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.991439] LustreError: 89369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.000835] LustreError: 89369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2539.603989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2539.898662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2543.632955] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2548.379833] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.423829] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2555.372331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2555.375166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.383550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2560.451596] LustreError: 90721:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2560.486651] LustreError: 88407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.633442] Lustre: server umount lustre-OST0000 complete [ 2565.607757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.611260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.613743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.628462] Lustre: Skipped 2 previous similar messages [ 2565.645751] Lustre: Skipped 1 previous similar message [ 2570.734826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2570.737398] Lustre: Skipped 1 previous similar message [ 2575.841193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.400134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2578.474847] LustreError: 90922:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2578.480768] LustreError: 90922:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2578.600554] Lustre: server umount lustre-MDT0000 complete [ 2580.976668] LustreError: 89372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.996264] LustreError: 89372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2582.102820] LustreError: 88402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344372 with bad export cookie 361858618832017936 [ 2582.104752] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.114082] LustreError: 88402:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2582.251619] LustreError: 91125:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2582.274819] LustreError: 91125:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2582.469743] Lustre: server umount lustre-MDT0001 complete [ 2591.151234] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2592.919942] Key type lgssc unregistered [ 2593.107442] LNet: 91657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.112835] LNetError: 91657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2594.155677] LNet: Removed LNI 192.168.202.153@tcp [ 2594.742073] Key type .llcrypt unregistered [ 2594.743890] Key type ._llcrypt unregistered [ 2615.489686] Key type ._llcrypt registered [ 2615.491258] Key type .llcrypt registered [ 2615.592351] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2616.361936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.387099] alg: No test for adler32 (adler32-zlib) [ 2617.384767] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2617.613856] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2619.328200] Key type lgssc registered [ 2620.373803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2627.610271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2627.635951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.495158] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2639.724308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.207877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2643.526692] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2645.475460] LustreError: 93717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.483399] LustreError: 93717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2650.471732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2650.595099] LustreError: 93723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.614413] LustreError: 93723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2650.752342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.666418] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2656.226245] LustreError: 93718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.237505] LustreError: 93718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2661.350575] LustreError: 93722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.404331] LustreError: 93722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2663.666213] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2663.692266] Lustre: MGS: Client cb6846d6-bcbc-4fcb-9e8c-bf39f972d2ca (at 0@lo) reconnecting [ 2663.918632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.009444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2667.403615] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2679.777976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.783165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.803289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2684.389350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2684.397288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.404082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.772792] LustreError: 95680:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2685.949134] Lustre: server umount lustre-OST0000 complete [ 2689.508655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2689.508982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.522163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.531244] Lustre: Skipped 1 previous similar message [ 2689.540904] Lustre: Skipped 2 previous similar messages [ 2694.629523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.636690] Lustre: Skipped 1 previous similar message [ 2699.746244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.748839] Lustre: Skipped 1 previous similar message [ 2703.328123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2703.456838] LustreError: 95883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2703.514498] LustreError: 95883:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2703.540353] Lustre: server umount lustre-MDT0000 complete [ 2704.880241] LustreError: 93719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.908376] LustreError: 93719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2707.979275] LustreError: 96088:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2707.981915] LustreError: 96088:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2708.222313] Lustre: server umount lustre-MDT0001 complete [ 2712.578341] LustreError: 96289:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2712.583124] LustreError: 96289:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2712.627557] Lustre: server umount MGS complete [ 2723.308744] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:55:11 (1763344511) [ 2743.345412] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 2751.172798] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2758.828147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2764.193735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.432943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2776.051342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.793683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.846502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.205196] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2793.660102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2793.707924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.903212] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2793.963418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2794.028416] Lustre: lustre-MDT0001: new disk, initializing [ 2794.089220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.114409] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2794.118242] Lustre: 100803:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2794.122362] LustreError: 100803:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2797.609642] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2807.525995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2807.608662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.822788] Lustre: lustre-OST0000: new disk, initializing [ 2807.838893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2807.916528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.678553] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2815.009782] Lustre: 101975:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005adaa6c4 x1849000624954624/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763344621 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2818.779733] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2825.101196] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2825.110161] Lustre: MGS: Client 9d74b034-f05c-4f5c-a0de-bb6996289358 (at 0@lo) reconnecting [ 2825.141811] Lustre: 100090:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2825.154365] LustreError: 100090:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2825.175632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2825.234867] Lustre: lustre-MDT0000: new disk, initializing [ 2825.383928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.409268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2825.493634] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2826.474337] Lustre: 102515:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e8151ec4 x1849000624960768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763344632 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2829.036127] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2831.540221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2831.561727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2831.629795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2852.321410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.325541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.336382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.589854] LustreError: 103042:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2857.592567] LustreError: 103042:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2857.723927] Lustre: server umount lustre-OST0000 complete [ 2861.538862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.541641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2861.552548] Lustre: Skipped 1 previous similar message [ 2861.565018] LustreError: Skipped 1 previous similar message [ 2866.486818] LustreError: 103245:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2866.496222] LustreError: 103245:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2866.528840] Lustre: server umount lustre-MDT0000 complete [ 2866.671747] LustreError: 100775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.712842] Lustre: server umount lustre-MDT0001 complete [ 2878.683585] Lustre: server umount MGS complete [ 2888.765354] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:57:56 (1763344676) [ 2903.236769] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2905.219472] Key type lgssc unregistered [ 2905.444907] LNet: 104984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.450679] LNetError: 104984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2905.466667] LNet: Removed LNI 192.168.202.153@tcp [ 2906.052590] Key type .llcrypt unregistered [ 2906.053852] Key type ._llcrypt unregistered [ 2921.439375] Key type ._llcrypt registered [ 2921.440712] Key type .llcrypt registered [ 2921.534309] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2922.472424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2922.529751] alg: No test for adler32 (adler32-zlib) [ 2923.622195] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2923.908535] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2925.608159] Key type lgssc registered [ 2926.961625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2934.924547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2934.943620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.347518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.162951] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2946.529728] LustreError: 106187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.539488] LustreError: 106187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2947.352583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2947.757392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2951.077909] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2957.241097] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2960.853035] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2966.717251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2967.218202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.731523] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2977.420481] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2981.564070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2989.029319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.037189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.057734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.100654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.104664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.106036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.222943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2997.231099] Lustre: Skipped 2 previous similar messages [ 3002.338771] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3002.346120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.406175] LustreError: 108434:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3002.550058] Lustre: server umount lustre-OST0000 complete [ 3007.461220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.467480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.470228] Lustre: Skipped 1 previous similar message [ 3012.389596] LustreError: 108638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3012.391809] LustreError: 108638:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3012.438827] Lustre: server umount lustre-MDT0000 complete [ 3012.581101] LustreError: 106183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.608439] LustreError: 106183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3016.394669] LustreError: 106905:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344806 with bad export cookie 16474211336283539224 [ 3016.400620] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.419500] LustreError: 106905:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3016.684969] LustreError: 108839:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3016.694499] LustreError: 108839:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3016.883105] Lustre: server umount lustre-MDT0001 complete [ 3025.638163] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3028.060174] Key type lgssc unregistered [ 3028.324813] LNet: 109372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3028.332950] LNetError: 109372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3028.343984] LNet: Removed LNI 192.168.202.153@tcp [ 3028.968640] Key type .llcrypt unregistered [ 3028.970400] Key type ._llcrypt unregistered [ 3041.812864] Key type ._llcrypt registered [ 3041.814889] Key type .llcrypt registered [ 3041.926741] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3042.660037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3042.689222] alg: No test for adler32 (adler32-zlib) [ 3043.727036] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3043.956627] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3045.672921] Key type lgssc registered [ 3046.951809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3052.419742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3052.427347] LustreError: 110409:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 3052.454233] LustreError: 110413:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 3052.457962] LustreError: 110409:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4973 [ 3052.461702] LustreError: 110409:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3052.476049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.485624] LustreError: 110409:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3053.962404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.857371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3062.274471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3066.702092] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3072.170683] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing lsmod [ 3075.580546] LustreError: 111597:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 3075.600569] LustreError: 111605:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 3075.606083] LustreError: 111597:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3075.616137] LustreError: 111605:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3075.622660] LustreError: 111605:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3075.627988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3076.192465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.196873] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.202338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.522769] LustreError: 111795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3085.677577] Lustre: server umount lustre-MDT0000 complete [ 3086.308501] LustreError: 110977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.327209] LustreError: 110977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3087.332103] LustreError: 111796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.354583] LustreError: 111796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3089.423753] LustreError: 110426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344879 with bad export cookie 17569189457960096888 [ 3089.424940] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3089.438201] LustreError: 110426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3089.461486] LustreError: 110886:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e5714420 x1849001072676992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3089.620095] LustreError: 111998:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3089.623094] LustreError: 111998:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3089.878967] Lustre: server umount lustre-MDT0001 complete [ 3104.018518] LustreError: 112199:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3104.034849] LustreError: 112199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3104.197575] Lustre: server umount lustre-OST0000 complete [ 3113.058495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.476584] LustreError: 112676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.551618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.253426] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3118.563447] LustreError: 112676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.657710] LustreError: 112676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.867683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3126.277492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3130.456673] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3135.572834] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3138.923920] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3145.839280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3146.223594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.137875] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3157.928236] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3158.527905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3166.689465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.694556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.707723] Lustre: Skipped 2 previous similar messages [ 3166.720128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3166.722904] Lustre: Skipped 2 previous similar messages [ 3168.737955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.742948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.745540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3170.373493] LustreError: 114924:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3170.375664] LustreError: 114924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3170.511626] Lustre: server umount lustre-OST0000 complete [ 3177.451834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3177.460907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.486376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.506098] Lustre: Skipped 1 previous similar message [ 3180.631496] LustreError: 115126:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3180.651775] LustreError: 115126:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3180.898296] Lustre: server umount lustre-MDT0000 complete [ 3184.107337] LustreError: 114040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3184.135396] LustreError: 114040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3184.951029] LustreError: 112656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344975 with bad export cookie 17569189457960097602 [ 3184.958779] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3184.969192] LustreError: 112656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3185.294867] Lustre: server umount lustre-MDT0001 complete [ 3193.735696] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3196.251135] Key type lgssc unregistered [ 3196.581908] LNet: 115860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3196.587839] LNetError: 115860:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3196.605687] LNet: Removed LNI 192.168.202.153@tcp [ 3197.277596] Key type .llcrypt unregistered [ 3197.279958] Key type ._llcrypt unregistered [ 3221.463983] Key type ._llcrypt registered [ 3221.466092] Key type .llcrypt registered [ 3221.596384] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3222.333163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3222.374957] alg: No test for adler32 (adler32-zlib) [ 3223.573253] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3223.892355] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3225.616185] Key type lgssc registered [ 3226.755391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3234.068970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3234.085746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.579376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.402988] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3245.029277] LustreError: 117257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.045600] LustreError: 117257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3246.895374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3247.165734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.139631] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3254.622974] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3258.430952] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3264.951799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3265.238070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.352815] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3275.174091] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3279.677625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3284.031655] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3286.928622] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3292.350770] LustreError: 119605:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3292.436666] Lustre: server umount lustre-OST0000 complete [ 3298.278449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3298.292947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.310118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.325093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.328938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.441174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.445984] Lustre: Skipped 2 previous similar messages [ 3306.491257] LustreError: 119809:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3306.494245] LustreError: 119809:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3306.542275] Lustre: server umount lustre-MDT0000 complete [ 3309.474136] Lustre: 116568:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763345083/real 1763345083] req@000000005b75bd7f x1849001261431296/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763345099 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3309.498550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.509250] LustreError: 117252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.519826] LustreError: 117252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3309.802613] LustreError: 117238:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345100 with bad export cookie 7120386493731247585 [ 3309.808196] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.817415] LustreError: 117238:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3309.982960] LustreError: 120011:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3309.988754] LustreError: 120011:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3310.178994] Lustre: server umount lustre-MDT0001 complete [ 3320.153643] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3322.802432] Key type lgssc unregistered [ 3323.063654] LNet: 120542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3323.073221] LNetError: 120542:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3323.084648] LNet: Removed LNI 192.168.202.153@tcp [ 3323.763990] Key type .llcrypt unregistered [ 3323.766388] Key type ._llcrypt unregistered [ 3338.564345] Key type ._llcrypt registered [ 3338.566281] Key type .llcrypt registered [ 3338.660766] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3339.844229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3339.854645] alg: No test for adler32 (adler32-zlib) [ 3341.107641] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3341.318671] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3342.976189] Key type lgssc registered [ 3344.170218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3351.579911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3351.602197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.112322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.970622] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3363.303052] LustreError: 121734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3363.307641] LustreError: 121734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3365.194684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3365.638855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3369.482577] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3374.913272] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3378.040993] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3384.075930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3384.357348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.370479] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3393.817674] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3397.686126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3407.856384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.859174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3407.872697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3407.877610] Lustre: Skipped 1 previous similar message [ 3412.819817] LustreError: 124003:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3413.005765] Lustre: server umount lustre-OST0000 complete [ 3417.065891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3417.076684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.117355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3417.120807] Lustre: Skipped 1 previous similar message [ 3418.084631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.694435] LustreError: 124204:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3422.697708] LustreError: 124204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3422.771495] Lustre: server umount lustre-MDT0000 complete [ 3423.206069] LustreError: 122765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.225168] LustreError: 122765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3425.957232] LustreError: 121714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345216 with bad export cookie 16158112990483370381 [ 3425.976199] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3426.106400] LustreError: 124406:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3426.112604] LustreError: 124406:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3426.271155] Lustre: server umount lustre-MDT0001 complete [ 3435.738947] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3438.551613] Key type lgssc unregistered [ 3438.913291] LNet: 124936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3438.930543] LNetError: 124936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3438.972947] LNet: Removed LNI 192.168.202.153@tcp [ 3439.871147] Key type .llcrypt unregistered [ 3439.872862] Key type ._llcrypt unregistered [ 3462.171825] Key type ._llcrypt registered [ 3462.173451] Key type .llcrypt registered [ 3462.267455] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3463.072737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.122264] alg: No test for adler32 (adler32-zlib) [ 3464.195574] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3464.421641] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3466.120180] Key type lgssc registered [ 3467.138322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3473.953802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.980852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.476613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.386267] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3484.650871] LustreError: 126335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.657967] LustreError: 126335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3486.713426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.985416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3489.963395] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3495.187518] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3498.293210] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3504.265146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3504.556813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.948712] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3514.938997] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3519.934475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3525.823236] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3529.512772] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3529.522411] Lustre: Skipped 2 previous similar messages [ 3532.912538] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3532.958422] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3534.205517] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3534.208695] Lustre: Skipped 2 previous similar messages [ 3541.588155] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3541.589787] Lustre: Skipped 1 previous similar message [ 3547.874495] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.53@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3547.883594] Lustre: Skipped 2 previous similar messages [ 3550.962374] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.53@tcp 192.168.0.[2,10]@tcp [ 3561.064918] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.53/24@tcp in log lustre-MDT0000 [ 3561.069929] Lustre: Skipped 2 previous similar messages [ 3568.170605] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.53/24@tcp [ 3568.185019] Lustre: Skipped 1 previous similar message [ 3574.764141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3574.764892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.781769] LustreError: Skipped 3 previous similar messages [ 3574.788602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3576.294112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3576.295149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.460081] LustreError: 130093:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3579.615997] Lustre: server umount lustre-OST0000 complete [ 3584.481323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3584.483847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3584.508434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3584.513377] Lustre: Skipped 1 previous similar message [ 3586.530411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.534450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.548170] Lustre: Skipped 1 previous similar message [ 3588.945473] LustreError: 130295:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3588.948289] LustreError: 130295:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3589.109674] Lustre: server umount lustre-MDT0000 complete [ 3591.650413] LustreError: 126335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.689739] LustreError: 126335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3592.834558] LustreError: 128634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345383 with bad export cookie 14129280665945348308 [ 3592.836645] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.851240] LustreError: 128634:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3592.990776] LustreError: 130497:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3592.999158] LustreError: 130497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3593.251924] Lustre: server umount lustre-MDT0001 complete [ 3603.573259] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3606.072274] Key type lgssc unregistered [ 3606.299443] LNet: 131029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3606.311380] LNetError: 131029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3606.327379] LNet: Removed LNI 192.168.202.153@tcp [ 3607.105929] Key type .llcrypt unregistered [ 3607.111421] Key type ._llcrypt unregistered [ 3633.110209] Key type ._llcrypt registered [ 3633.111467] Key type .llcrypt registered [ 3633.214947] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3633.950109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.979774] alg: No test for adler32 (adler32-zlib) [ 3635.007931] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3635.212638] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3636.912161] Key type lgssc registered [ 3638.021972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3645.494939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3645.514168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.096432] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3647.127455] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.53/24@tcp [ 3647.129519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.189623] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3656.677618] LustreError: 132571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.706297] LustreError: 132571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3660.394258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.837251] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3660.860965] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.53/24@tcp [ 3660.951562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3665.594329] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3670.746371] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3674.507110] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3683.126486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3683.571777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.625267] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3691.769842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3697.002503] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3712.484317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.487985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.488256] LustreError: Skipped 1 previous similar message [ 3712.489348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.530962] Lustre: Skipped 1 previous similar message [ 3714.323688] LustreError: 134873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3714.493531] Lustre: server umount lustre-OST0000 complete [ 3722.213031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3722.230233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.245917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.254399] Lustre: Skipped 1 previous similar message [ 3724.428837] LustreError: 135075:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3724.460629] LustreError: 135075:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3724.568306] Lustre: server umount lustre-MDT0000 complete [ 3727.850258] LustreError: 132568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.866808] LustreError: 132568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3728.607924] LustreError: 132554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345518 with bad export cookie 3913931417794125646 [ 3728.616980] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3728.620507] LustreError: 132554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3728.736581] LustreError: 135277:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3728.754542] LustreError: 135277:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3729.006838] Lustre: server umount lustre-MDT0001 complete [ 3742.738822] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3745.491905] Key type lgssc unregistered [ 3745.784552] LNet: 135810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3745.789326] LNetError: 135810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3746.859417] LNet: Removed LNI 192.168.202.153@tcp [ 3747.818223] Key type .llcrypt unregistered [ 3747.820695] Key type ._llcrypt unregistered [ 3774.039840] Key type ._llcrypt registered [ 3774.041573] Key type .llcrypt registered [ 3774.201876] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3788.178893] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3789.514965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3789.553545] alg: No test for adler32 (adler32-zlib) [ 3790.627796] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3790.829535] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3792.496263] Key type lgssc registered [ 3793.684570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3803.472861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3810.794182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3817.348420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3823.861265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3830.063779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3830.117374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3830.129076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.409942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3831.470939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3831.565551] Lustre: lustre-MDT0000: new disk, initializing [ 3831.638411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.653072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3836.168321] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3845.113156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3845.175937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3845.237385] Lustre: 139189:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3845.294250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3845.297777] Lustre: Skipped 1 previous similar message [ 3845.350907] Lustre: lustre-MDT0001: new disk, initializing [ 3845.430944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.460943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3845.466036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3848.605051] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3855.157453] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3858.352620] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3864.825569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3864.890992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3865.208398] Lustre: lustre-OST0000: new disk, initializing [ 3865.223124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3865.323610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.414254] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3874.917471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3874.931746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3875.090295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3877.170825] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3881.865159] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3882.069704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3886.516037] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3886.691133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3890.659290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.666391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3890.671050] Lustre: Skipped 1 previous similar message [ 3890.680239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3894.530081] LustreError: 142125:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3894.716252] Lustre: server umount lustre-OST0000 complete [ 3900.898241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.899986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.905921] Lustre: Skipped 1 previous similar message [ 3900.919518] Lustre: Skipped 1 previous similar message [ 3906.018955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.023337] Lustre: Skipped 1 previous similar message [ 3911.138900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.184109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3913.327856] LustreError: 142327:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3913.334777] LustreError: 142327:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3913.499326] Lustre: server umount lustre-MDT0000 complete [ 3917.211425] LustreError: 139968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345707 with bad export cookie 10291205241472149086 [ 3917.219212] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3917.221835] LustreError: 139968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3917.381447] LustreError: 142531:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3917.384028] LustreError: 142531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3917.565903] Lustre: server umount lustre-MDT0001 complete [ 3924.748100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.128303] LustreError: 143007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3925.162876] LustreError: 143007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3925.227652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.863840] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3930.593271] LustreError: 143007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.697988] LustreError: 143007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.386367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.878154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3941.701570] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3946.518787] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3949.933560] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3956.523912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3956.882417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.825726] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3968.272935] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3971.847061] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3972.086541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3975.717427] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3975.917694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3983.690537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3983.766445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3983.878138] Lustre: lustre-OST0001: new disk, initializing [ 3983.880942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3983.977697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.947983] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3992.149928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3992.162172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3992.208442] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3998.000853] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3998.211132] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4002.092660] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4002.299511] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4005.643071] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f32c6485000.ost_server_uuid 50 [ 4007.129292] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f32c6485000.ost_server_uuid in FULL state after 0 sec [ 4018.150139] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.153706] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4018.202059] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4018.204774] Lustre: Skipped 1 previous similar message [ 4021.027721] LustreError: 147254:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4021.029849] LustreError: 147254:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4021.151612] Lustre: server umount lustre-OST0001 complete [ 4023.269816] LustreError: 144606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.310829] LustreError: 144606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4024.800947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4024.807368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.814552] Lustre: Skipped 1 previous similar message [ 4026.850249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4026.863279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.872616] Lustre: Skipped 1 previous similar message [ 4028.400491] LustreError: 144605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.406532] LustreError: 144605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4038.629248] LustreError: 144606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.670688] LustreError: 144606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4039.140230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4039.212248] LustreError: 147455:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4039.215176] LustreError: 147455:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4039.386934] Lustre: server umount lustre-OST0000 complete [ 4043.749273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.759134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.769332] Lustre: Skipped 2 previous similar messages [ 4043.799683] Lustre: Skipped 7 previous similar messages [ 4049.159134] LustreError: 147659:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4049.245163] LustreError: 147659:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4049.374353] Lustre: server umount lustre-MDT0000 complete [ 4053.281568] LustreError: 146020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345843 with bad export cookie 10291205241472149975 [ 4053.284883] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4053.308339] LustreError: 146020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4054.021280] Lustre: server umount lustre-MDT0001 complete [ 4073.132808] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4075.967539] Key type lgssc unregistered [ 4076.327545] LNet: 148849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4076.338280] LNetError: 148849:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4076.356198] LNet: Removed LNI 192.168.202.153@tcp [ 4077.195882] Key type .llcrypt unregistered [ 4077.206285] Key type ._llcrypt unregistered [ 4111.031131] Key type ._llcrypt registered [ 4111.041095] Key type .llcrypt registered [ 4111.188363] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4126.157494] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4127.238130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4127.303117] alg: No test for adler32 (adler32-zlib) [ 4128.435801] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4128.615716] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4130.301896] Key type lgssc registered [ 4131.593978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4141.585212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4148.717991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.901589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4161.553942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.329663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.406723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4167.425051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.648245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4168.673728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4168.744627] Lustre: lustre-MDT0000: new disk, initializing [ 4168.842067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.880080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4172.570583] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4183.313713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.392184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4183.483573] Lustre: 152803:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4183.574592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4183.578732] Lustre: Skipped 1 previous similar message [ 4183.705194] Lustre: lustre-MDT0001: new disk, initializing [ 4183.871874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4183.939744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4183.952650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4188.330136] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4196.288762] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4200.593169] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4206.481135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4206.532210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4206.694921] Lustre: lustre-OST0000: new disk, initializing [ 4206.699683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4206.802864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.864404] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4215.898422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4215.912554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4216.054098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4219.458597] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4223.280234] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4223.523290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4227.035973] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4227.211997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4231.651721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.660881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.665528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.680230] Lustre: Skipped 1 previous similar message [ 4231.726803] Lustre: Skipped 1 previous similar message [ 4234.951380] LustreError: 155745:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4235.099407] Lustre: server umount lustre-OST0000 complete [ 4240.355474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4240.370296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.397615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.891621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.894607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.800519] LustreError: 155947:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4244.807783] LustreError: 155947:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4244.934614] Lustre: server umount lustre-MDT0000 complete [ 4248.120603] LustreError: 152794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346038 with bad export cookie 7930425814423471781 [ 4248.121749] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.135498] LustreError: 152794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.356392] LustreError: 156149:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4248.365396] LustreError: 156149:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4248.605568] Lustre: server umount lustre-MDT0001 complete [ 4255.627658] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:20:43 (1763346043) [ 4269.408953] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4280.108226] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4289.911264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4296.725575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4302.702788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4308.355897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.026594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.105801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.302893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4314.344184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4314.427942] Lustre: lustre-MDT0000: new disk, initializing [ 4314.519781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.550622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4317.965823] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4328.096513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4328.204364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4328.341506] Lustre: 159142:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4328.390499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4328.393394] Lustre: Skipped 1 previous similar message [ 4328.452513] Lustre: lustre-MDT0001: new disk, initializing [ 4328.548768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4328.583547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4328.588151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4332.120521] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4338.775521] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4341.853353] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4347.152781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4347.192504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4347.373729] Lustre: lustre-OST0000: new disk, initializing [ 4347.386716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4347.473984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.059116] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4353.582394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4353.592389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4353.699560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4360.370627] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4364.441710] Lustre: Failing over lustre-OST0000 [ 4366.551641] LustreError: 161585:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4366.578364] LustreError: 161585:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4366.759798] Lustre: server umount lustre-OST0000 complete [ 4382.689171] Lustre: 151014:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346157/real 1763346157] req@00000000b655962d x1849002209383040/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1763346173 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4382.735748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.254954] Lustre: 151013:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346159/real 1763346159] req@000000009a597ee9 x1849002209383808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4386.042630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4386.336448] Lustre: 151014:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346159/real 1763346159] req@00000000c1e98b13 x1849002209384320/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763346175 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4386.384770] Lustre: 151014:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4386.650693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.675710] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4389.729376] Lustre: 151012:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346164/real 1763346164] req@000000009985f747 x1849002209385344/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4389.796950] Lustre: 151012:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4392.830133] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4393.001554] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4394.941708] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4394.944250] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4396.001182] Lustre: 151013:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346169/real 1763346169] req@00000000bc1450d9 x1849002209386368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4396.553535] Lustre: Failing over lustre-MDT0000 [ 4396.692722] LustreError: 162566:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4396.704974] LustreError: 162566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4396.828533] Lustre: server umount lustre-MDT0000 complete [ 4397.923137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.927284] Lustre: Skipped 1 previous similar message [ 4397.939353] LustreError: 159153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.975710] LustreError: 159153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4398.846450] LustreError: 160043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.170698] LustreError: 160043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.209809] LustreError: 160043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4408.289593] LustreError: 159154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.296722] LustreError: 159154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4413.416026] LustreError: 161171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.448327] LustreError: 161171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4414.438437] Lustre: 151013:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346188/real 1763346188] req@00000000b3405635 x1849002209398656/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763346204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4414.472619] Lustre: 151013:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4414.496655] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.408624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.653194] LustreError: 159154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.680413] LustreError: 159154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 4423.949045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.009612] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4425.314043] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4427.486389] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4429.293715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4429.298943] Lustre: Skipped 1 previous similar message [ 4429.322631] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4429.359620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4434.914470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4434.924988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.941464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4434.944479] Lustre: Skipped 1 previous similar message [ 4436.680257] LustreError: 163454:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4436.694133] LustreError: 163454:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4436.841624] Lustre: server umount lustre-OST0000 complete [ 4444.643902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4444.644382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.658077] LustreError: Skipped 1 previous similar message [ 4444.658557] Lustre: Skipped 1 previous similar message [ 4446.401173] Lustre: server umount lustre-MDT0000 complete [ 4449.559589] LustreError: 159133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346239 with bad export cookie 7930425814423473895 [ 4449.564464] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4449.571307] LustreError: 159133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4449.957354] Lustre: server umount lustre-MDT0001 complete [ 4459.590308] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4461.987962] Key type lgssc unregistered [ 4462.246361] LNet: 164388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4462.258826] LNetError: 164388:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4462.275627] LNet: Removed LNI 192.168.202.153@tcp [ 4462.944685] Key type .llcrypt unregistered [ 4462.946545] Key type ._llcrypt unregistered [ 4496.623813] Key type ._llcrypt registered [ 4496.626165] Key type .llcrypt registered [ 4496.711879] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4497.442266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4497.469702] alg: No test for adler32 (adler32-zlib) [ 4498.549939] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4498.827692] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4500.480169] Key type lgssc registered [ 4501.467332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4507.607756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4507.627663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.016876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.485454] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4519.397282] LustreError: 165891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.418622] LustreError: 165891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4519.447354] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4523.232976] LustreError: 165919:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4523.241138] LustreError: 165919:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4523.245694] LustreError: 165919:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4523.356740] LustreError: 166442:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4523.551748] Lustre: server umount lustre-MDT0000 complete [ 4538.672161] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4551.453949] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4562.727872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4569.928205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4575.781480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4581.635122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4588.300335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4588.379684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.587704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.626148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4588.700321] Lustre: lustre-MDT0000: new disk, initializing [ 4588.818231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.847047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4592.754453] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4602.699965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4602.816039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4602.916806] Lustre: 169088:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4602.978556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4602.983153] Lustre: Skipped 1 previous similar message [ 4603.039830] Lustre: lustre-MDT0001: new disk, initializing [ 4603.138390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4603.184023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4603.194557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4606.338200] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4613.453329] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4617.056949] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4623.285863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4623.353929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4623.574093] Lustre: lustre-OST0000: new disk, initializing [ 4623.590696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4623.664955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.232650] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4629.594658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4629.602605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4629.687186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4634.551181] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4637.712520] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4639.078864] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:27:07 (1763346427) [ 4865.505417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.506915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4865.520253] Lustre: Skipped 2 previous similar messages [ 4870.625953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4870.630126] Lustre: Skipped 4 previous similar messages [ 4871.495761] LustreError: 172611:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4871.507428] LustreError: 172611:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4871.650826] Lustre: server umount lustre-MDT0000 complete [ 4875.044801] LustreError: 169082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346665 with bad export cookie 10226219291521906106 [ 4875.048942] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4875.074144] LustreError: 169082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4875.744836] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.746626] LustreError: 172491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.758280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4880.864780] LustreError: 170216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.865069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4881.267174] LustreError: 172814:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4881.270206] LustreError: 172814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4881.349726] Lustre: server umount lustre-MDT0001 complete [ 4890.573051] LustreError: 173015:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4890.593247] LustreError: 173015:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4890.707179] Lustre: server umount lustre-OST0000 complete [ 4900.614788] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4912.502729] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4921.755215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4928.144994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4934.462995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4940.712895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4948.218122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4948.361261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.670053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4948.726398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4948.844674] Lustre: lustre-MDT0000: new disk, initializing [ 4948.973340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.991749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4952.212633] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4960.998911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4961.063954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4961.148381] Lustre: 175360:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4961.229439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4961.233756] Lustre: Skipped 1 previous similar message [ 4961.311744] Lustre: lustre-MDT0001: new disk, initializing [ 4961.413664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4961.449884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4961.465585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4965.192366] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4972.082594] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4976.017063] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4982.738825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4982.803913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4983.110898] Lustre: lustre-OST0000: new disk, initializing [ 4983.116758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4983.297405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.602782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4990.548510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4990.560590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4990.691711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4997.574220] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5002.046425] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5002.268484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5006.679770] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5006.990879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5011.432251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5011.433287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5011.453419] LustreError: Skipped 1 previous similar message [ 5011.459827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5011.487040] Lustre: Skipped 1 previous similar message [ 5021.668628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.671767] Lustre: Skipped 2 previous similar messages [ 5023.200231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5023.243314] LustreError: 178296:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5023.257918] LustreError: 178296:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5023.417862] Lustre: server umount lustre-OST0000 complete [ 5026.790216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.801314] Lustre: Skipped 2 previous similar messages [ 5041.120133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5041.240313] LustreError: 178499:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5041.248024] LustreError: 178499:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5041.331233] Lustre: server umount lustre-MDT0000 complete [ 5042.148956] LustreError: 175366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.167308] LustreError: 175366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5044.463321] LustreError: 175351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346834 with bad export cookie 10226219291522064936 [ 5044.467288] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5044.469624] LustreError: 175351:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5044.754610] Lustre: server umount lustre-MDT0001 complete [ 5052.126268] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:33:59 (1763346839) [ 5069.413147] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 5077.558651] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5086.977709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5094.380393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5100.983730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5107.530406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5114.428538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5114.492085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.834240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5114.873228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5114.943738] Lustre: lustre-MDT0000: new disk, initializing [ 5115.019212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.039714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5119.514697] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5130.364963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.466077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5130.610178] Lustre: 181697:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.974738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5135.669743] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5144.099329] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5148.315979] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5155.596123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5155.707485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5156.055221] Lustre: lustre-OST0000: new disk, initializing [ 5156.060755] Lustre: Skipped 1 previous similar message [ 5156.074551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5156.096100] Lustre: Skipped 2 previous similar messages [ 5162.221986] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5164.105023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5164.112582] Lustre: Skipped 1 previous similar message [ 5164.115232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5164.252869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5170.022952] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5179.878947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.882402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5179.883277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5179.883282] Lustre: Skipped 7 previous similar messages [ 5179.892183] Lustre: Skipped 1 previous similar message [ 5179.899662] LustreError: Skipped 1 previous similar message [ 5182.590137] LustreError: 184244:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5182.607493] LustreError: 184244:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5182.689797] Lustre: server umount lustre-OST0000 complete [ 5187.043397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5190.115974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5190.130654] Lustre: Skipped 1 previous similar message [ 5200.353715] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5200.752647] Lustre: server umount lustre-MDT0000 complete [ 5204.517718] LustreError: 181688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346994 with bad export cookie 10226219291522065825 [ 5204.529590] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.873545] Lustre: server umount lustre-MDT0001 complete [ 5213.777689] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:36:41 (1763347001) [ 5229.066689] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 5239.509343] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5249.033119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5254.937444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5261.174592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5267.240077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.497281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.576893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.837831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5273.872375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5273.936410] Lustre: lustre-MDT0000: new disk, initializing [ 5274.032676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.036023] Lustre: Skipped 2 previous similar messages [ 5274.048247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5278.107677] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5288.516879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5288.582840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5288.648825] Lustre: 187643:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5288.931451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5293.297678] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5300.790563] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5304.273527] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5310.578435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5310.652208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5316.341616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5316.493471] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5323.965874] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5331.943795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.948247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5331.987133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5332.006394] Lustre: Skipped 9 previous similar messages [ 5337.730317] LustreError: 190189:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5337.739054] LustreError: 190189:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5337.859421] Lustre: server umount lustre-OST0000 complete [ 5348.313743] Lustre: server umount lustre-MDT0000 complete [ 5352.419760] LustreError: 188842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5352.446199] LustreError: 188842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5352.585841] LustreError: 187635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347142 with bad export cookie 10226219291522066910 [ 5352.594042] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5352.602702] LustreError: 187635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5352.900801] Lustre: server umount lustre-MDT0001 complete [ 5364.966570] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5367.832756] Key type lgssc unregistered [ 5368.128681] LNet: 191125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.142385] LNetError: 191125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.151867] LNet: Removed LNI 192.168.202.153@tcp [ 5368.916631] Key type .llcrypt unregistered [ 5368.921664] Key type ._llcrypt unregistered [ 5392.017289] Key type ._llcrypt registered [ 5392.019047] Key type .llcrypt registered [ 5392.164001] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5393.300410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5393.366097] alg: No test for adler32 (adler32-zlib) [ 5394.484245] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5394.724382] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5396.376307] Key type lgssc registered [ 5397.547132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5404.626135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5404.643938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.135911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.825208] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5416.420868] LustreError: 192525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.437569] LustreError: 192525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5417.727761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5417.985441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5421.213079] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5426.365919] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5430.406921] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5438.653488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5438.909091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5444.800981] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5450.995420] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5451.199766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5471.713447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.727666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5471.739084] Lustre: Skipped 1 previous similar message [ 5476.321180] LustreError: 194796:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5476.481635] Lustre: server umount lustre-OST0000 complete [ 5481.958232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.969880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5481.975808] Lustre: Skipped 1 previous similar message [ 5486.451807] LustreError: 194997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5486.468160] LustreError: 194997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5486.566376] Lustre: server umount lustre-MDT0000 complete [ 5487.073718] LustreError: 192524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.106265] LustreError: 192524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5490.414827] LustreError: 192506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347280 with bad export cookie 8459890013507291107 [ 5490.430524] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5490.430922] LustreError: 192506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5490.602541] LustreError: 195199:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5490.605837] LustreError: 195199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5490.837556] Lustre: server umount lustre-MDT0001 complete [ 5503.931960] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5506.967083] Key type lgssc unregistered [ 5507.254782] LNet: 195732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5507.263711] LNetError: 195732:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5507.289242] LNet: Removed LNI 192.168.202.153@tcp [ 5508.047996] Key type .llcrypt unregistered [ 5508.049714] Key type ._llcrypt unregistered [ 5533.987036] Key type ._llcrypt registered [ 5533.988814] Key type .llcrypt registered [ 5534.146188] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5535.102144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5535.158786] alg: No test for adler32 (adler32-zlib) [ 5536.380708] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5536.586602] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5538.280206] Key type lgssc registered [ 5539.444584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5548.660456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5548.709184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.296887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5554.841721] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5559.789570] LustreError: 197134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.837504] LustreError: 197134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5564.111177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5564.449677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.195438] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5574.537427] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5578.355263] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5585.137876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5585.543827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5591.521825] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5594.805602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5597.322898] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5605.352382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.355356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.357597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5605.376266] Lustre: Skipped 1 previous similar message [ 5609.996174] LustreError: 199380:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5610.044095] Lustre: server umount lustre-OST0000 complete [ 5615.975561] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f32c4769000.ost_server_uuid 50 [ 5617.884976] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5635.046235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.059183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.078515] Lustre: Skipped 1 previous similar message [ 5635.085506] Lustre: Skipped 2 previous similar messages [ 5638.551746] LustreError: 199748:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5638.559886] LustreError: 199748:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5638.684309] Lustre: server umount lustre-MDT0000 complete [ 5640.165580] LustreError: 197133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.194681] LustreError: 197133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5642.495111] LustreError: 197115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347432 with bad export cookie 16936884591937971104 [ 5642.503750] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.517787] LustreError: 197115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5642.651041] LustreError: 199950:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5642.654167] LustreError: 199950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5642.918902] Lustre: server umount lustre-MDT0001 complete [ 5651.739348] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:43:59 (1763347439) [ 5658.252740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5658.620536] LustreError: 200622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.694419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5662.542873] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5663.720253] LustreError: 200622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.837965] LustreError: 200621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.546215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5670.970613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.702920] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5680.808790] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5685.392221] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5692.422335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5692.723440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.972912] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5703.273457] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5703.992870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5709.972643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5710.041735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5710.155845] Lustre: lustre-OST0001: new disk, initializing [ 5710.159257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5710.267917] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5716.230902] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5719.638517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5719.647417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5719.717695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5724.397074] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5729.763547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5729.775185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5729.794736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5733.080474] LustreError: 203829:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5733.088826] LustreError: 203829:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5733.176138] Lustre: server umount lustre-OST0000 complete [ 5735.142312] LustreError: 203143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.047614] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5739.242443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5742.873186] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5743.083400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5755.878209] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.879721] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.883395] LustreError: 203143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.883405] LustreError: 203143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5755.883603] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5755.883606] Lustre: Skipped 3 previous similar messages [ 5755.893430] Lustre: Skipped 2 previous similar messages [ 5760.996548] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5761.012693] Lustre: Skipped 2 previous similar messages [ 5761.540768] LustreError: 204563:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5761.542633] LustreError: 204563:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5761.664107] Lustre: server umount lustre-OST0001 complete [ 5766.115485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.237304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5771.250526] Lustre: Skipped 3 previous similar messages [ 5771.423084] LustreError: 204764:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5771.426217] LustreError: 204764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5771.657048] Lustre: server umount lustre-MDT0000 complete [ 5775.626657] LustreError: 200602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347566 with bad export cookie 16936884591937972175 [ 5775.631733] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5775.636425] LustreError: 200602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5776.078180] Lustre: server umount lustre-MDT0001 complete [ 5780.928445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5784.553786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5788.621363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5792.619783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.431694] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:46:26 (1763347586) [ 5804.668269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.143224] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5805.187783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5805.395865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.300773] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5817.446424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5817.547094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5817.592320] Lustre: 206461:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5821.699877] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5827.690666] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5831.874938] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5838.140020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5838.262642] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5844.178084] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5846.666633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5850.675316] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5857.817284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5857.872190] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5864.794147] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5870.442526] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5874.665081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.679669] Lustre: Skipped 1 previous similar message [ 5874.683671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5874.686663] Lustre: Skipped 1 previous similar message [ 5879.178223] LustreError: 209658:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5879.180693] LustreError: 209658:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5879.243464] Lustre: server umount lustre-OST0000 complete [ 5879.786702] LustreError: 208569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5879.808076] LustreError: 208569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5894.627856] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.633056] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.638257] Lustre: Skipped 1 previous similar message [ 5900.363764] Lustre: server umount lustre-OST0001 complete [ 5904.865200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5904.877702] LustreError: Skipped 1 previous similar message [ 5909.986721] Lustre: server umount lustre-MDT0000 complete [ 5913.917695] LustreError: 206451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347704 with bad export cookie 16936884591937973582 [ 5913.926400] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.936655] LustreError: 206451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5914.137862] LustreError: 210286:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5914.139868] LustreError: 210286:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5914.334817] Lustre: server umount lustre-MDT0001 complete [ 5920.312424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5924.507601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5928.200903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5931.774717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5938.561742] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:48:46 (1763347726) [ 5952.591528] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 5960.337321] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5968.668592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5974.392591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5979.658662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5985.469906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5991.123838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5991.184187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5991.351835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5991.391093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5991.449584] Lustre: lustre-MDT0000: new disk, initializing [ 5991.535248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.538226] Lustre: Skipped 3 previous similar messages [ 5991.561237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5994.657759] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6004.666975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6004.743156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6004.838293] Lustre: 214110:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6004.884493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6004.886848] Lustre: Skipped 1 previous similar message [ 6004.934536] Lustre: lustre-MDT0001: new disk, initializing [ 6005.074091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6005.097906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6008.346622] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6015.242804] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6019.012837] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6024.903909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6024.952437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6025.186810] Lustre: lustre-OST0000: new disk, initializing [ 6025.190500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6025.201377] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6025.214196] Lustre: Skipped 7 previous similar messages [ 6030.332977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6030.340028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6030.404813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6031.435794] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6039.661550] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6044.049532] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6044.310277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6048.832339] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6049.115412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6055.917606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.927930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.935990] Lustre: Skipped 4 previous similar messages [ 6055.964055] LustreError: Skipped 1 previous similar message [ 6057.174145] LustreError: 217053:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6057.186548] LustreError: 217053:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6057.385118] Lustre: server umount lustre-OST0000 complete [ 6067.538050] Lustre: server umount lustre-MDT0000 complete [ 6070.694603] LustreError: 215854:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347861 with bad export cookie 16936884591937975143 [ 6070.698965] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6070.703346] LustreError: 215854:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6071.235172] Lustre: server umount lustre-MDT0001 complete [ 6078.473889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6078.877562] LustreError: 217944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.893086] LustreError: 217944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 6082.246396] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6090.921518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6095.077168] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6101.039916] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6105.428269] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6112.022878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6118.429525] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6123.642071] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6127.739795] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6127.917432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6132.006221] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6132.252355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6137.319496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6137.329630] LustreError: Skipped 1 previous similar message [ 6137.332161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.342850] Lustre: Skipped 2 previous similar messages [ 6137.348859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6137.351717] Lustre: Skipped 5 previous similar messages [ 6140.625745] Lustre: server umount lustre-OST0000 complete [ 6146.329598] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6150.743770] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6178.076664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6178.306590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.311365] Lustre: Skipped 5 previous similar messages [ 6179.306920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.322245] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6182.836906] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6187.618707] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6189.545479] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.551223] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6198.862637] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 6204.900678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6204.903955] LustreError: Skipped 1 previous similar message [ 6207.550160] LustreError: 222348:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6207.552839] LustreError: 222348:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6207.653346] Lustre: server umount lustre-OST0000 complete [ 6216.876363] Lustre: server umount lustre-MDT0000 complete [ 6220.258399] LustreError: 217945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.264022] LustreError: 217945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6220.305561] LustreError: 217924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348010 with bad export cookie 16936884591937976025 [ 6220.315440] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6220.319576] LustreError: 217924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6220.672057] Lustre: server umount lustre-MDT0001 complete [ 6228.474465] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:53:36 (1763348016) [ 6234.321204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.904203] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6247.158747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6251.116942] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6255.720243] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6258.932211] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6264.332708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6269.231837] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6273.952202] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6277.916782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6280.304210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6284.254116] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6284.475767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6291.183985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6291.233790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6291.368972] Lustre: lustre-OST0001: new disk, initializing [ 6291.376940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6296.250464] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6296.638237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6296.652994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6296.744701] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6302.246864] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6305.397548] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6305.582146] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6308.430716] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6308.651896] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6312.423196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.427392] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.428116] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6312.428124] Lustre: Skipped 7 previous similar messages [ 6312.440501] LustreError: Skipped 1 previous similar message [ 6312.490371] Lustre: Skipped 6 previous similar messages [ 6316.537713] Lustre: server umount lustre-OST0001 complete [ 6321.882850] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6322.063344] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6325.250339] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6325.465143] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6353.904071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6359.536974] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6359.553769] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6359.563869] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6360.518211] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6366.150708] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6367.672938] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6375.798306] Lustre: server umount lustre-OST0001 complete [ 6397.310946] Lustre: server umount lustre-OST0000 complete [ 6406.711655] Lustre: server umount lustre-MDT0000 complete [ 6409.951861] LustreError: 229720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348200 with bad export cookie 16936884591937976991 [ 6409.964735] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.468093] Lustre: server umount lustre-MDT0001 complete [ 6422.319645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6426.422182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.339548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.600490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6441.739867] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:57:09 (1763348229) [ 6448.420394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6448.710849] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6448.769158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6448.980300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.984340] Lustre: Skipped 5 previous similar messages [ 6452.535419] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6460.611558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6460.669794] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6460.693857] Lustre: 231415:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6464.579456] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6470.036486] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6474.192598] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6480.599364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6480.840067] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6486.262156] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6490.958543] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6500.028311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6500.115982] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6505.537132] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6510.867365] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6512.428798] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6516.352898] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6516.578256] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6520.489815] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6520.678394] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6524.798549] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f32c800d000.ost_server_uuid 50 [ 6526.305489] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f32c800d000.ost_server_uuid in FULL state after 0 sec [ 6527.454365] Lustre: Permanently deactivating lustre-OST0001 [ 6527.487203] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6530.500460] Lustre: Permanently reactivating lustre-OST0001 [ 6536.682279] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6536.685238] LustreError: Skipped 5 previous similar messages [ 6539.771176] LustreError: 235403:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6539.782876] LustreError: 235403:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6539.828392] Lustre: server umount lustre-OST0001 complete [ 6541.799156] LustreError: 233516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6541.807778] LustreError: 233516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 6557.157870] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6557.272267] Lustre: server umount lustre-OST0000 complete [ 6566.558607] Lustre: server umount lustre-MDT0000 complete [ 6570.082935] LustreError: 232154:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348360 with bad export cookie 16936884591937978272 [ 6570.088277] LustreError: 232154:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6570.093407] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.403310] Lustre: server umount lustre-MDT0001 complete [ 6575.336429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.100540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6582.552409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6585.657407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6591.255911] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:59:39 (1763348379) [ 6592.123516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.410846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.689229] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6597.742613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6597.754811] Lustre: Skipped 5 previous similar messages [ 6601.408705] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6608.587335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6608.678829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6608.714803] Lustre: 237564:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6612.532499] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6617.902596] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6621.114515] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6626.418791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6626.479502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6626.611066] Lustre: Permanently deactivating lustre-OST0000 [ 6626.625448] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6630.598440] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6633.595553] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6635.561800] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6641.962947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6642.012207] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6642.015051] Lustre: Skipped 1 previous similar message [ 6643.600366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6647.508618] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6653.305163] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6655.352950] Lustre: Permanently reactivating lustre-OST0000 [ 6655.371977] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6655.374297] Lustre: Skipped 2 previous similar messages [ 6665.294989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.304993] Lustre: Skipped 12 previous similar messages [ 6665.310564] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp 000000002948e817, cur 1763348455 deadline 1763348454 last 1763348424 [ 6665.317935] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6665.322070] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6665.326598] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6665.329308] Lustre: Skipped 1 previous similar message [ 6665.380652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6670.833680] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6670.848612] Lustre: Skipped 26 previous similar messages [ 6671.967657] Lustre: server umount lustre-OST0001 complete [ 6681.858374] Lustre: server umount lustre-OST0000 complete [ 6691.283668] Lustre: server umount lustre-MDT0000 complete [ 6694.585426] LustreError: 237556:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348484 with bad export cookie 16936884591937979959 [ 6694.589098] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.600384] LustreError: 237556:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6694.900695] Lustre: server umount lustre-MDT0001 complete [ 6704.282819] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6706.778454] Key type lgssc unregistered [ 6707.047334] LNet: 241993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6707.061317] LNetError: 241993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6708.073837] LNet: Removed LNI 192.168.202.153@tcp [ 6708.958162] Key type .llcrypt unregistered [ 6708.961801] Key type ._llcrypt unregistered [ 6728.648533] Key type ._llcrypt registered [ 6728.650378] Key type .llcrypt registered [ 6728.730512] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6729.594324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6729.640421] alg: No test for adler32 (adler32-zlib) [ 6730.656761] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 6730.832481] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6732.488321] Key type lgssc registered [ 6733.257716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6735.622711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6739.699419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6739.707239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.003635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.564433] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6750.329962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6750.401867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6750.474474] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6750.498798] Lustre: 243407:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6750.512968] Lustre: Permanently deactivating lustre-MDT0001 [ 6750.538144] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6750.881197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6754.389932] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6758.408903] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6759.441573] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6762.815939] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6768.229694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6768.516075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.322638] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6775.652224] LustreError: 245043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.680806] LustreError: 245043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6778.667762] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6780.915787] LustreError: 245042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.732094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6785.940725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6786.995616] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6786.999514] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6787.020304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6791.874532] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6796.779732] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6799.869649] Lustre: Permanently reactivating lustre-MDT0001 [ 6799.891078] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6799.914930] Lustre: Skipped 2 previous similar messages [ 6803.038761] Lustre: lustre-MDT0001: Client e1e706d6-de5a-496c-a3c1-9a5934645189 (at 192.168.202.53@tcp) reconnecting [ 6806.697691] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6806.721129] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6806.748926] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6806.759249] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6808.074139] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6810.041165] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6815.786557] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6815.999298] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6818.943961] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6819.111502] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6821.266397] Lustre: Permanently deactivating lustre-MDT0001 [ 6821.296772] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6821.299648] Lustre: Skipped 2 previous similar messages [ 6830.127161] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6837.356455] LustreError: 247512:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6837.438449] Lustre: server umount lustre-MDT0000 complete [ 6837.733058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6837.736743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.744879] LustreError: 244150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.521413] LustreError: 243400:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348630 with bad export cookie 7284012228018200160 [ 6840.524790] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6840.535196] LustreError: 243400:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6840.663754] LustreError: 247713:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6840.667736] LustreError: 247713:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6840.811795] Lustre: server umount lustre-MDT0001 complete [ 6844.032165] LustreError: 247915:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6844.034860] LustreError: 247915:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6844.095024] Lustre: server umount lustre-OST0000 complete [ 6854.532661] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:04:02 (1763348642) [ 6863.813516] LustreError: 248920:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6863.816143] LustreError: 248920:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6863.919369] Lustre: server umount lustre-OST0001 complete [ 6868.796181] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 6877.886727] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6887.089578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6892.999868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6897.714127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6902.762431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6908.049702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6908.106477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.281454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6908.289598] Lustre: Skipped 2 previous similar messages [ 6908.307032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6908.397053] Lustre: lustre-MDT0000: new disk, initializing [ 6908.466397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.485390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6911.100561] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6918.602367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6918.648838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6918.709378] Lustre: 251107:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6918.760448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6918.770018] Lustre: Skipped 1 previous similar message [ 6918.844439] Lustre: lustre-MDT0001: new disk, initializing [ 6918.964397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6919.004643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6919.011575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6922.833985] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6929.758483] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6933.306779] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6939.398453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.463899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6939.656677] Lustre: lustre-OST0000: new disk, initializing [ 6939.659695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6939.752038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6944.037902] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6946.374830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6946.390474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6946.474988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6950.733126] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6952.593769] LustreError: 252349:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6959.370930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6959.434622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6959.541114] Lustre: lustre-OST0001: new disk, initializing [ 6959.545498] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6962.656226] LustreError: 252349:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6962.690549] LustreError: 251112:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6964.242669] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6966.866568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6966.880692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6966.946815] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6970.846622] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6972.531260] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6972.703037] LustreError: 251112:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6975.738488] Lustre: lustre-OST0001: Not available for connect from 192.168.202.53@tcp (stopping) [ 6977.507879] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.514456] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6977.546763] Lustre: Skipped 4 previous similar messages [ 6977.551424] Lustre: Skipped 1 previous similar message [ 6980.223534] LustreError: 254600:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6980.228144] LustreError: 254600:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6980.299834] Lustre: server umount lustre-OST0001 complete [ 6980.835249] LustreError: 252837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.850262] LustreError: 252837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6987.751916] LustreError: 253301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.759496] LustreError: 253301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6997.995239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.996224] LustreError: 252836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6997.998851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6997.999260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6998.015499] Lustre: Skipped 1 previous similar message [ 6998.030044] LustreError: 252836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7003.111886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.114716] Lustre: Skipped 1 previous similar message [ 7008.235785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7008.250461] Lustre: Skipped 2 previous similar messages [ 7009.248163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7009.310217] LustreError: 254824:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7009.313032] LustreError: 254824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7009.538828] Lustre: server umount lustre-OST0000 complete [ 7013.346121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.356554] Lustre: Skipped 1 previous similar message [ 7018.467406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7018.470183] Lustre: Skipped 2 previous similar messages [ 7019.753854] Lustre: server umount lustre-MDT0000 complete [ 7023.311535] LustreError: 251097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348813 with bad export cookie 7284012228018202764 [ 7023.313243] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7023.318860] LustreError: 251097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7029.476300] LustreError: 255228:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7029.481539] LustreError: 255228:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7029.601336] Lustre: server umount lustre-MDT0001 complete [ 7038.788988] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7041.131633] Key type lgssc unregistered [ 7041.436758] LNet: 255762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7041.443748] LNetError: 255762:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7041.457881] LNet: Removed LNI 192.168.202.153@tcp [ 7042.041079] Key type .llcrypt unregistered [ 7042.045273] Key type ._llcrypt unregistered [ 7074.378073] Key type ._llcrypt registered [ 7074.380240] Key type .llcrypt registered [ 7074.453938] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 7090.273276] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7091.112823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7091.226039] alg: No test for adler32 (adler32-zlib) [ 7092.422593] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 7092.637034] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7094.320174] Key type lgssc registered [ 7095.503407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.345202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7112.374933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7117.961892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7123.752448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7131.675994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7131.769075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7131.802752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7133.180215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7133.211702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7133.336719] Lustre: lustre-MDT0000: new disk, initializing [ 7133.458364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.503205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7138.472812] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7149.237527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7149.355699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7149.572158] Lustre: 259713:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7149.691737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7149.701595] Lustre: Skipped 1 previous similar message [ 7149.794348] Lustre: lustre-MDT0001: new disk, initializing [ 7149.921912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.955692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7149.959677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7154.079644] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7160.789607] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7164.380848] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7169.995778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7170.060555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7170.294468] Lustre: lustre-OST0000: new disk, initializing [ 7170.305352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7170.360054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.005763] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7176.825724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7176.840955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7177.047417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7183.428984] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7187.035443] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7187.248287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7190.563825] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7190.732779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7192.547932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7192.550807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.576233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7192.585761] Lustre: Skipped 1 previous similar message [ 7192.601729] Lustre: Skipped 1 previous similar message [ 7197.668409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.422220] LustreError: 262650:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7198.665941] Lustre: server umount lustre-OST0000 complete [ 7202.792857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.803670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7202.806421] Lustre: Skipped 1 previous similar message [ 7207.909112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.916138] Lustre: Skipped 1 previous similar message [ 7208.502840] LustreError: 262852:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7208.509727] LustreError: 262852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7208.733270] Lustre: server umount lustre-MDT0000 complete [ 7212.291178] LustreError: 259704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349002 with bad export cookie 1410879187119297818 [ 7212.294237] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7212.315846] LustreError: 259704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7212.550720] LustreError: 263055:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7212.572317] LustreError: 263055:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7212.807606] Lustre: server umount lustre-MDT0001 complete [ 7222.012755] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 22:10:09 (1763349009) [ 7227.813771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.325529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7232.288341] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7233.508065] LustreError: 263727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.529633] LustreError: 263727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7238.625263] LustreError: 263727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.693244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7240.999098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7244.844906] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7250.325359] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7254.042349] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7260.295367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7260.533257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7266.052920] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7272.513727] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7325.153465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.156872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.205967] Lustre: Skipped 1 previous similar message [ 7325.235831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.249032] Lustre: Skipped 1 previous similar message [ 7329.286640] LustreError: 266057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7329.296361] LustreError: 266057:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7329.576342] Lustre: server umount lustre-OST0000 complete [ 7334.368081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7347.603735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7347.983096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.941901] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7360.233245] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7365.351193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7365.374665] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7365.378861] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7401.444356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.447223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7401.454689] Lustre: Skipped 1 previous similar message [ 7401.457600] LustreError: Skipped 1 previous similar message [ 7401.458265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.845151] LustreError: 267721:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7406.864033] LustreError: 267721:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7406.988448] Lustre: server umount lustre-OST0000 complete [ 7411.683520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.688128] Lustre: Skipped 1 previous similar message [ 7416.558432] LustreError: 267922:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7416.563427] LustreError: 267922:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7416.660267] Lustre: server umount lustre-MDT0000 complete [ 7416.803360] LustreError: 263722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.823646] LustreError: 263722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7419.692842] LustreError: 263706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349210 with bad export cookie 1410879187119298700 [ 7419.700820] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.703784] LustreError: 263706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7420.057666] Lustre: server umount lustre-MDT0001 complete [ 7430.233235] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7432.853843] Key type lgssc unregistered [ 7433.175774] LNet: 268657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7433.185929] LNetError: 268657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7433.201698] LNet: Removed LNI 192.168.202.153@tcp [ 7434.067097] Key type .llcrypt unregistered [ 7434.068546] Key type ._llcrypt unregistered [ 7456.564575] Key type ._llcrypt registered [ 7456.566158] Key type .llcrypt registered [ 7456.713395] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7458.022836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7458.051865] alg: No test for adler32 (adler32-zlib) [ 7459.088333] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 7459.374788] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7461.040177] Key type lgssc registered [ 7462.047611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7468.700334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7468.739402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.281283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.077232] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7479.780689] LustreError: 270056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7479.790273] LustreError: 270056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7481.823511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7482.140105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7486.120307] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7490.772871] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7493.877252] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7500.217663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7500.554338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.718882] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7511.214256] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7514.930399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7535.589573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.597363] Lustre: Skipped 1 previous similar message [ 7535.597911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7538.945116] LustreError: 273073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7539.029354] Lustre: server umount lustre-OST0000 complete [ 7543.776952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7543.786306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.801246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.803402] Lustre: Skipped 1 previous similar message [ 7545.825485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.827277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7548.273912] LustreError: 273276:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7548.326064] LustreError: 273276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7548.398269] Lustre: server umount lustre-MDT0000 complete [ 7550.946464] LustreError: 270053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.966581] LustreError: 270053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7551.547587] LustreError: 270036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349341 with bad export cookie 11818756941402901813 [ 7551.552606] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7551.555278] LustreError: 270036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7551.670356] LustreError: 273477:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7551.673689] LustreError: 273477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7551.861360] Lustre: server umount lustre-MDT0001 complete [ 7559.715476] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7562.044545] Key type lgssc unregistered [ 7562.299676] LNet: 274010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7562.308291] LNetError: 274010:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7562.321797] LNet: Removed LNI 192.168.202.153@tcp [ 7562.929190] Key type .llcrypt unregistered [ 7562.930977] Key type ._llcrypt unregistered [ 7577.583692] Key type ._llcrypt registered [ 7577.585279] Key type .llcrypt registered [ 7577.654611] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7578.335348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7578.455484] alg: No test for adler32 (adler32-zlib) [ 7579.487609] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 7579.663560] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7581.324266] Key type lgssc registered [ 7582.470418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7590.074142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7590.098442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.507374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.423453] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7601.638635] LustreError: 275222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.659255] LustreError: 275222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7603.734850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7604.047813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7607.610445] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7612.580383] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7616.303937] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7622.671727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7622.965976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7628.350375] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7633.892059] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7637.305347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7649.765702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7649.775662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.794880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7652.847796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7652.853296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7652.881971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.917775] LustreError: 277614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7655.054344] Lustre: server umount lustre-OST0000 complete [ 7660.514190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7660.517694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7660.532549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7660.535596] Lustre: Skipped 1 previous similar message [ 7663.076290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.076494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.090725] Lustre: Skipped 1 previous similar message [ 7668.195577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.202517] Lustre: Skipped 1 previous similar message [ 7672.288431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7672.359842] LustreError: 277816:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7672.375281] LustreError: 277816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7672.459217] Lustre: server umount lustre-MDT0000 complete [ 7673.328276] LustreError: 277158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.345320] LustreError: 277158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7675.304166] LustreError: 275202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349465 with bad export cookie 15342462088227747536 [ 7675.309525] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7675.312475] LustreError: 275202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7675.443282] LustreError: 278018:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7675.451778] LustreError: 278018:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7675.610982] Lustre: server umount lustre-MDT0001 complete [ 7682.908809] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7684.667967] Key type lgssc unregistered [ 7684.845180] LNet: 278550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7684.849490] LNetError: 278550:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7684.860167] LNet: Removed LNI 192.168.202.153@tcp [ 7685.385289] Key type .llcrypt unregistered [ 7685.387897] Key type ._llcrypt unregistered [ 7703.190955] Key type ._llcrypt registered [ 7703.192893] Key type .llcrypt registered [ 7703.266447] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7703.965023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7704.034589] alg: No test for adler32 (adler32-zlib) [ 7704.984029] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 7705.139099] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7706.824209] Key type lgssc registered [ 7707.844271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7714.773940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7714.794630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7716.296343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.811726] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7726.580234] LustreError: 279926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.629103] LustreError: 279926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7727.686829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7728.033799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7731.398813] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7736.456382] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7740.348194] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7747.032035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7747.414725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.590802] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7760.959601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7761.173711] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7786.466035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7786.467650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.519037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.005384] LustreError: 282947:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7790.195919] Lustre: server umount lustre-OST0000 complete [ 7794.657196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7794.660313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.672725] Lustre: Skipped 1 previous similar message [ 7794.677710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.686761] Lustre: Skipped 1 previous similar message [ 7796.708426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.714618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.733348] Lustre: Skipped 1 previous similar message [ 7799.656724] LustreError: 283149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7799.659497] LustreError: 283149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7799.886948] Lustre: server umount lustre-MDT0000 complete [ 7801.825190] LustreError: 279926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7801.862835] LustreError: 279926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7803.790229] LustreError: 279906:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349594 with bad export cookie 5569075835134406438 [ 7803.810747] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7803.946151] LustreError: 283350:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7803.948547] LustreError: 283350:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7804.164846] Lustre: server umount lustre-MDT0001 complete [ 7814.246380] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7817.114043] Key type lgssc unregistered [ 7817.459714] LNet: 283882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7817.472860] LNetError: 283882:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7817.498722] LNet: Removed LNI 192.168.202.153@tcp [ 7818.475861] Key type .llcrypt unregistered [ 7818.477644] Key type ._llcrypt unregistered [ 7836.590183] Key type ._llcrypt registered [ 7836.591866] Key type .llcrypt registered [ 7836.729990] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7837.768786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7837.910286] alg: No test for adler32 (adler32-zlib) [ 7838.976392] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 7839.292924] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7841.017724] Key type lgssc registered [ 7842.396817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7850.771325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7850.829783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7852.499863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.549820] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7861.736458] LustreError: 285110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.753242] LustreError: 285110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7865.387388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7865.702447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7869.453200] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7875.368105] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7879.697440] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7886.443163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7886.959515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7892.438371] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7897.210529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7898.016546] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7911.905585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7911.911315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.928408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7912.937130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.948158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7914.112184] LustreError: 287517:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7914.225604] Lustre: server umount lustre-OST0000 complete [ 7918.049681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.060603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7918.083033] Lustre: Skipped 1 previous similar message [ 7918.084840] Lustre: Skipped 1 previous similar message [ 7923.171565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7923.183861] Lustre: Skipped 2 previous similar messages [ 7928.293154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7928.310474] Lustre: Skipped 1 previous similar message [ 7932.387015] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7932.471575] LustreError: 287718:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7932.476843] LustreError: 287718:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7932.624498] Lustre: server umount lustre-MDT0000 complete [ 7933.414151] LustreError: 285106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.447144] LustreError: 285106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7936.535384] LustreError: 285065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349726 with bad export cookie 12043190671540476517 [ 7936.547844] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7936.740404] LustreError: 287921:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7936.744448] LustreError: 287921:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7936.910184] Lustre: server umount lustre-MDT0001 complete [ 7949.476664] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7952.514493] Key type lgssc unregistered [ 7952.890696] LNet: 288452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7952.902158] LNetError: 288452:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7952.925131] LNet: Removed LNI 192.168.202.153@tcp [ 7953.827282] Key type .llcrypt unregistered [ 7953.828901] Key type ._llcrypt unregistered [ 7966.089585] Key type ._llcrypt registered [ 7966.091131] Key type .llcrypt registered [ 7966.211473] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7983.764135] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 7998.847763] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8000.304297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8000.315727] alg: No test for adler32 (adler32-zlib) [ 8001.404583] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 8001.612871] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8003.288424] Key type lgssc registered [ 8004.567865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8012.771422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8019.413599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8025.491790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8032.199117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8038.320752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8038.374153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8038.388130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.642980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8039.678153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8039.760549] Lustre: lustre-MDT0000: new disk, initializing [ 8039.869748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8039.899247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8043.466341] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8053.755250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8053.823632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8053.948560] Lustre: 291964:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8054.016697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8054.020702] Lustre: Skipped 1 previous similar message [ 8054.108877] Lustre: lustre-MDT0001: new disk, initializing [ 8054.218317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8054.283987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8054.297484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8057.689899] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8064.344834] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8067.701632] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8073.613570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8073.663122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8073.832985] Lustre: lustre-OST0000: new disk, initializing [ 8073.835629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8073.936361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8078.880872] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8080.457653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8080.472851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8080.631486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8086.817430] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8091.339519] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8091.637157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8095.573548] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8095.852370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8101.345150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8101.351866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.355887] LustreError: Skipped 1 previous similar message [ 8101.360097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8101.369598] Lustre: Skipped 1 previous similar message [ 8101.382734] Lustre: Skipped 1 previous similar message [ 8104.066388] LustreError: 294901:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8104.231679] Lustre: server umount lustre-OST0000 complete [ 8110.561519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8110.564934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8110.602212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8111.594587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.607709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8111.612304] Lustre: Skipped 1 previous similar message [ 8116.715369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.722524] Lustre: Skipped 1 previous similar message [ 8120.802182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8120.810641] Lustre: Skipped 1 previous similar message [ 8122.336142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8122.458430] LustreError: 295103:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8122.462566] LustreError: 295103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8122.586890] Lustre: server umount lustre-MDT0000 complete [ 8126.074510] LustreError: 291956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349916 with bad export cookie 1229163617906838360 [ 8126.094708] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8126.108131] LustreError: 291956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8126.345030] LustreError: 295305:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8126.360088] LustreError: 295305:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8126.558869] Lustre: server umount lustre-MDT0001 complete [ 8136.040968] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:25:23 (1763349923) [ 8143.753784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.383995] LustreError: 295978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.409981] LustreError: 295978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8144.468752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8148.353436] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8149.483417] LustreError: 295978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8154.593633] LustreError: 295978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8157.120358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8157.455348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8160.962718] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8166.694886] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8170.781379] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8178.596256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8179.068109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.137670] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8190.584104] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8199.650955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8199.653673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8199.666362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8205.182110] LustreError: 298225:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8205.186158] LustreError: 298225:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8205.367745] Lustre: server umount lustre-OST0000 complete [ 8209.892144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8209.901924] Lustre: Skipped 2 previous similar messages [ 8220.130183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.132721] Lustre: Skipped 7 previous similar messages [ 8223.202431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8223.337636] LustreError: 298427:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8223.354941] LustreError: 298427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8223.491587] Lustre: server umount lustre-MDT0000 complete [ 8225.261699] LustreError: 295972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.279918] LustreError: 295972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8227.984530] LustreError: 295959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350018 with bad export cookie 1229163617906839242 [ 8228.000210] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8228.008094] LustreError: 295959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8228.429447] Lustre: server umount lustre-MDT0001 complete [ 8240.582459] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8243.352482] Key type lgssc unregistered [ 8243.649477] LNet: 299166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8243.665916] LNetError: 299166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8244.711390] LNet: Removed LNI 192.168.202.153@tcp [ 8245.530424] Key type .llcrypt unregistered [ 8245.532171] Key type ._llcrypt unregistered [ 8275.361575] Key type ._llcrypt registered [ 8275.365441] Key type .llcrypt registered [ 8275.515512] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8292.252892] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8293.464862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8293.487129] alg: No test for adler32 (adler32-zlib) [ 8294.690446] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 8294.856141] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8296.496189] Key type lgssc registered [ 8297.686979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8306.246294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8312.888854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8319.618157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8326.178146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8331.722358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8337.315961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8343.827871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8343.888474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8343.899789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8345.101537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8345.132837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8345.190534] Lustre: lustre-MDT0000: new disk, initializing [ 8345.316984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8345.351941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8349.228727] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8358.677189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8358.731196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8358.801128] Lustre: 302978:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8358.866692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8358.869064] Lustre: Skipped 1 previous similar message [ 8358.934585] Lustre: lustre-MDT0001: new disk, initializing [ 8359.026247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8359.059165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8359.064464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8362.979884] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8370.129410] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8373.886464] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8381.102600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8381.186493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8381.498799] Lustre: lustre-OST2710: new disk, initializing [ 8381.503312] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8381.601270] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8386.564144] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8390.704546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8390.722876] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8390.809258] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8393.577179] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8400.757435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8400.826324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8400.939704] Lustre: lustre-OST03e8: new disk, initializing [ 8400.944936] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8401.013904] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8406.105923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8406.118400] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8406.199082] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8406.587364] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8412.107993] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8419.775203] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8419.963428] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8425.243369] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8425.571436] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8430.519218] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8430.829458] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8436.273657] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8436.600745] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8442.347818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.350838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.354461] Lustre: Skipped 1 previous similar message [ 8442.368030] Lustre: Skipped 3 previous similar messages [ 8443.257752] LustreError: 307321:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8443.640834] Lustre: server umount lustre-MDT0000 complete [ 8448.402112] LustreError: 307322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350238 with bad export cookie 944285784165231027 [ 8448.405476] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.445811] LustreError: 307322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8448.756559] LustreError: 307523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8448.759322] LustreError: 307523:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8449.302080] Lustre: server umount lustre-MDT0001 complete [ 8464.005571] LustreError: 307724:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8464.008551] LustreError: 307724:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8464.119585] Lustre: server umount lustre-OST2710 complete [ 8467.940611] Lustre: 300759:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350242/real 1763350242] req@000000007f36b246 x1849006578791936/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763350258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8467.967634] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.974425] Lustre: Skipped 2 previous similar messages [ 8468.202179] LustreError: 307926:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8468.205309] LustreError: 307926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8468.433555] Lustre: server umount lustre-OST03e8 complete [ 8485.786880] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 8495.669655] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8504.809469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8511.890162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8518.578118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8525.134743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8531.434433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8531.479041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.668733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8531.687922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8531.750456] Lustre: lustre-MDT0000: new disk, initializing [ 8531.825186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.842168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8535.399985] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8545.190654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8545.246665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8545.310588] Lustre: 310725:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8545.359928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8545.363341] Lustre: Skipped 1 previous similar message [ 8545.438591] Lustre: lustre-MDT0001: new disk, initializing [ 8545.516762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8545.556203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8545.562800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8549.098945] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8556.702372] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8560.841840] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8569.083432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8569.129578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8569.392661] Lustre: lustre-OST0000: new disk, initializing [ 8569.406614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8569.491858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.655309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8575.670989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8575.793312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8576.808443] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8585.180403] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8590.207876] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8590.438408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8594.037400] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8594.263367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8596.457545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.457686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8596.462697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.470801] Lustre: Skipped 1 previous similar message [ 8596.480903] LustreError: Skipped 1 previous similar message [ 8601.576261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8601.580635] Lustre: Skipped 2 previous similar messages [ 8606.691528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8606.694115] Lustre: Skipped 1 previous similar message [ 8610.272895] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8610.326680] LustreError: 313668:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8610.335028] LustreError: 313668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8610.484922] Lustre: server umount lustre-OST0000 complete [ 8616.932028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8616.938136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8616.947034] Lustre: Skipped 1 previous similar message [ 8616.953855] Lustre: Skipped 1 previous similar message [ 8627.172093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8627.174735] Lustre: Skipped 2 previous similar messages [ 8628.192158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8628.261512] LustreError: 313871:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8628.267235] LustreError: 313871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8628.352654] Lustre: server umount lustre-MDT0000 complete [ 8631.589352] LustreError: 310716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350421 with bad export cookie 944285784165232700 [ 8631.592236] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8631.610145] LustreError: 310716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8631.881259] Lustre: server umount lustre-MDT0001 complete [ 8640.161723] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:33:48 (1763350428) [ 8641.806791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8643.296631] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:33:51 (1763350431) [ 8645.717949] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8652.533654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8656.226318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8659.957164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8663.304949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8664.457659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8671.242618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8671.505977] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8671.530485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8671.738346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.258632] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8683.600826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8683.778448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8683.798939] Lustre: 316207:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8688.162105] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8693.869613] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8697.195352] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8703.653571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8703.730486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8703.909963] Lustre: Denying initial registration attempt from nid 192.168.202.153@tcp, specified as failover [ 8703.915960] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8703.921600] LustreError: 317821:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8703.925989] LustreError: 317821:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8703.929096] LustreError: 317821:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8703.997238] Lustre: server umount lustre-OST0000 complete [ 8703.999556] LustreError: 317821:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8710.629472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8710.643155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8710.645769] Lustre: Skipped 1 previous similar message [ 8716.505333] LustreError: 318189:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8716.509901] LustreError: 318189:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8716.653764] Lustre: server umount lustre-MDT0000 complete [ 8720.436501] LustreError: 316197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350510 with bad export cookie 944285784165233589 [ 8720.441147] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.451653] LustreError: 316197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8720.768377] Lustre: server umount lustre-MDT0001 complete [ 8729.187895] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8731.507701] Key type lgssc unregistered [ 8731.729277] LNet: 318924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8731.735223] LNetError: 318924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8731.748593] LNet: Removed LNI 192.168.202.153@tcp [ 8732.379989] Key type .llcrypt unregistered [ 8732.381785] Key type ._llcrypt unregistered [ 8742.526647] Key type ._llcrypt registered [ 8742.529398] Key type .llcrypt registered [ 8742.644322] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8743.321020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.425102] alg: No test for adler32 (adler32-zlib) [ 8744.416866] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 8744.569112] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8746.192163] Key type lgssc registered [ 8747.144290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8753.235534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8756.547895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8759.933676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8763.877618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8765.060407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8780.397649] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8786.950386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8786.963521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.210691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8788.233771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8788.356682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.342845] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8798.332283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8798.427212] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8798.455739] Lustre: 321682:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8798.637946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8801.612394] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8806.637354] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8810.049303] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8816.989057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8817.061564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8817.245343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8817.429783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8822.000970] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8826.250674] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8828.386201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8828.390638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.402247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8828.903126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8828.911128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.926912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8829.949156] LustreError: 323967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8830.092934] Lustre: server umount lustre-OST0000 complete [ 8834.019476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.021644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8834.029542] Lustre: Skipped 1 previous similar message [ 8834.038494] Lustre: Skipped 1 previous similar message [ 8839.140345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.144608] Lustre: Skipped 1 previous similar message [ 8844.259963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.352938] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8848.459893] LustreError: 324168:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8848.473292] LustreError: 324168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8848.554614] Lustre: server umount lustre-MDT0000 complete [ 8852.150283] LustreError: 321673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350642 with bad export cookie 9882602386974230069 [ 8852.158987] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.188078] LustreError: 321673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8852.323346] LustreError: 324371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8852.339632] LustreError: 324371:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8852.559214] Lustre: server umount lustre-MDT0001 complete [ 8862.566311] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 8864.839782] Key type lgssc unregistered [ 8865.177655] LNet: 324902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8865.186575] LNetError: 324902:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8865.203835] LNet: Removed LNI 192.168.202.153@tcp [ 8865.988099] Key type .llcrypt unregistered [ 8865.994471] Key type ._llcrypt unregistered [ 8888.411466] Key type ._llcrypt registered [ 8888.413142] Key type .llcrypt registered [ 8888.547243] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 8889.479791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8889.510791] alg: No test for adler32 (adler32-zlib) [ 8890.555742] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 8890.730589] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 8892.378083] Key type lgssc registered [ 8893.296419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8900.376782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8900.399786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.862951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8905.139691] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8912.355960] LustreError: 326301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.371601] LustreError: 326301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8912.614548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8912.951793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8917.005584] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8921.767464] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8925.399644] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8931.191447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8931.521387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8936.385508] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 8940.644765] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8953.827490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8953.831436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.840909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.411822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.411854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.534754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8962.551306] Lustre: Skipped 5 previous similar messages [ 8967.650384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8967.657497] Lustre: Skipped 3 previous similar messages [ 8968.160112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8968.249747] LustreError: 328575:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8968.334800] Lustre: server umount lustre-MDT0000 complete [ 8970.952947] LustreError: 326281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350761 with bad export cookie 9821494431526118954 [ 8970.967139] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8972.769949] LustreError: 327317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8972.771025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.783683] Lustre: Skipped 1 previous similar message [ 8972.787205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8972.792624] Lustre: Skipped 1 previous similar message [ 8975.778935] LustreError: 326296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8975.787966] LustreError: 326296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8977.145345] LustreError: 328778:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8977.160970] LustreError: 328778:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8977.454022] Lustre: server umount lustre-MDT0001 complete [ 8983.670776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8991.213907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.330571] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x884cf526229784c3 [ 8996.364779] Lustre: MGC192.168.202.153@tcp: Connection restored to 0@lo (at 0@lo) [ 8996.600077] LustreError: 327912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8996.666882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8996.714300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9000.743915] LustreError: 329586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9000.787223] LustreError: 329586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9000.902472] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9005.875929] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9009.371763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9009.713936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9012.994746] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9014.759384] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9016.930287] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9020.625681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 9024.105417] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9025.763990] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9030.114280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9030.115205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.123903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9030.128380] Lustre: Skipped 1 previous similar message [ 9030.131134] Lustre: Skipped 2 previous similar messages [ 9035.400848] LustreError: 331081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9035.405108] LustreError: 331081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9035.572728] Lustre: server umount lustre-OST0000 complete [ 9040.353822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.355197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9045.294347] LustreError: 331283:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9045.303234] LustreError: 331283:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9045.381336] Lustre: server umount lustre-MDT0000 complete [ 9045.475859] LustreError: 329591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.483219] LustreError: 329591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9048.091074] LustreError: 326281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350838 with bad export cookie 9821494431526126787 [ 9048.102696] LustreError: 326281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9048.106109] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9048.314183] Lustre: server umount lustre-MDT0001 complete [ 9055.678100] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9057.938904] Key type lgssc unregistered [ 9058.201825] LNet: 332017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9058.218182] LNetError: 332017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9058.235311] LNet: Removed LNI 192.168.202.153@tcp [ 9058.823413] Key type .llcrypt unregistered [ 9058.824546] Key type ._llcrypt unregistered [ 9081.095447] Key type ._llcrypt registered [ 9081.097067] Key type .llcrypt registered [ 9081.190421] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9081.977119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9082.030909] alg: No test for adler32 (adler32-zlib) [ 9083.041534] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 9083.224370] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 9084.880183] Key type lgssc registered [ 9085.937972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9093.399587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9093.421842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.856272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.152511] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9105.383288] LustreError: 333415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9105.411542] LustreError: 333415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9106.180873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9106.589569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9110.017955] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9114.189798] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9116.988603] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9121.256978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9121.264845] Lustre: Skipped 1 previous similar message [ 9121.268460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9121.279545] Lustre: Skipped 1 previous similar message [ 9126.371771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9131.494102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9131.496848] Lustre: Skipped 2 previous similar messages [ 9134.048163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9134.128923] LustreError: 334805:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9134.238068] Lustre: server umount lustre-MDT0000 complete [ 9136.610423] LustreError: 333415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9136.618589] LustreError: 333415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9137.328154] LustreError: 333394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350927 with bad export cookie 3503931653356142557 [ 9137.330367] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9137.340964] LustreError: 333394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9137.493202] LustreError: 335008:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9137.496274] LustreError: 335008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9137.668969] Lustre: server umount lustre-MDT0001 complete [ 9149.675162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9149.909606] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9149.942621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9150.162099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9153.291659] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9160.059159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9160.184986] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9160.218756] Lustre: 335779:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9160.446958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9163.918416] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9169.727319] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9172.433677] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9178.657900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9178.825388] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9178.835203] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9178.841398] LustreError: 337436:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 9178.847225] LustreError: 337436:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 9178.851952] LustreError: 337436:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9178.891553] Lustre: server umount lustre-OST0000 complete [ 9178.894123] LustreError: 337436:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9186.789992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9187.148060] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9187.340669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.889234] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9195.190891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9198.389641] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9205.493803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9205.571649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9205.653864] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9205.732236] Lustre: lustre-OST0001: new disk, initializing [ 9205.735686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9205.835316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9210.426363] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9215.429418] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9215.562211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9215.575258] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9215.693772] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9221.095917] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9221.098408] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.100450] LustreError: Skipped 1 previous similar message [ 9221.101067] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9221.111766] Lustre: Skipped 1 previous similar message [ 9224.062130] LustreError: 339560:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9224.072090] LustreError: 339560:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9224.145995] Lustre: server umount lustre-OST0001 complete [ 9226.214505] LustreError: 337905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9226.226247] LustreError: 337905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9230.820184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9230.828955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.842750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9230.845767] Lustre: Skipped 1 previous similar message [ 9231.339297] LustreError: 337904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.345850] LustreError: 337904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9233.214155] LustreError: 339761:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9233.222817] LustreError: 339761:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9233.342957] Lustre: server umount lustre-OST0000 complete [ 9236.451124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.459845] Lustre: Skipped 2 previous similar messages [ 9241.569225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.574127] Lustre: Skipped 5 previous similar messages [ 9250.272137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9250.347347] LustreError: 339963:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9250.351476] LustreError: 339963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9250.410335] Lustre: server umount lustre-MDT0000 complete [ 9251.809239] LustreError: 335786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9251.823373] LustreError: 335786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9253.190597] LustreError: 335770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351043 with bad export cookie 3503931653356143110 [ 9253.197502] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.200732] LustreError: 335770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9253.452203] Lustre: server umount lustre-MDT0001 complete [ 9261.266246] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9263.069328] Key type lgssc unregistered [ 9263.330544] LNet: 340701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9263.340748] LNetError: 340701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9263.356832] LNet: Removed LNI 192.168.202.153@tcp [ 9263.892424] Key type .llcrypt unregistered [ 9263.894262] Key type ._llcrypt unregistered [ 9289.184552] Key type ._llcrypt registered [ 9289.186246] Key type .llcrypt registered [ 9289.299477] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9302.128783] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9303.246108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9303.257394] alg: No test for adler32 (adler32-zlib) [ 9304.307882] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 9304.528887] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 9306.160321] Key type lgssc registered [ 9307.147129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9314.094915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9319.137037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9324.240607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9329.582777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9334.716821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9334.777039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9334.793552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.015727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9336.040854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9336.103746] Lustre: lustre-MDT0000: new disk, initializing [ 9336.199704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.215939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9339.171924] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9345.763571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9345.809678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9345.865305] Lustre: 344650:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9345.902538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9345.905679] Lustre: Skipped 1 previous similar message [ 9345.952174] Lustre: lustre-MDT0001: new disk, initializing [ 9346.005847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9346.065226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9346.073528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9348.352688] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9353.862441] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9356.377753] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9361.386744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9361.429756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9361.578505] Lustre: lustre-OST0000: new disk, initializing [ 9361.582670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9361.639435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.215463] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9370.715530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9370.728270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9370.789795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9371.168171] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9374.757559] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9374.986954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9378.971799] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9379.215879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9381.348633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9381.353534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.365973] LustreError: Skipped 1 previous similar message [ 9381.374780] Lustre: Skipped 1 previous similar message [ 9381.379134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9386.468055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9386.476049] Lustre: Skipped 2 previous similar messages [ 9391.586351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9391.589128] Lustre: Skipped 1 previous similar message [ 9395.168148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9395.223154] LustreError: 347586:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9395.384320] Lustre: server umount lustre-OST0000 complete [ 9401.825453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.832302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.836600] Lustre: Skipped 1 previous similar message [ 9401.851450] Lustre: Skipped 1 previous similar message [ 9405.559741] LustreError: 347790:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9405.568321] LustreError: 347790:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9405.757632] Lustre: server umount lustre-MDT0000 complete [ 9409.182928] LustreError: 344643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351199 with bad export cookie 5547999545161424660 [ 9409.189744] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9409.191615] LustreError: 344643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9409.327607] LustreError: 347991:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9409.332818] LustreError: 347991:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9409.540771] Lustre: server umount lustre-MDT0001 complete [ 9417.414802] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 22:46:45 (1763351205) [ 9421.709884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9426.042322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9443.863854] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 9451.491395] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9459.541865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9465.533496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9470.762366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9477.131561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9483.291145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9483.340152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.525628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9483.553052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9483.613064] Lustre: lustre-MDT0000: new disk, initializing [ 9483.683042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.693123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9487.230227] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9498.242351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9498.314188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9498.372720] Lustre: 351768:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9498.420436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9498.424976] Lustre: Skipped 1 previous similar message [ 9498.499238] Lustre: lustre-MDT0001: new disk, initializing [ 9498.575249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9498.601630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9498.614786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9501.592911] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9508.223961] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9511.530513] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9517.828580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9517.902486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9518.118423] Lustre: lustre-OST0000: new disk, initializing [ 9518.123456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9518.190224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9522.825785] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9526.341408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9526.354865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9526.409336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9529.014778] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9533.342693] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9533.589994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9536.798579] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9536.935313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9542.120138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9542.122470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9542.124526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9542.134445] LustreError: Skipped 1 previous similar message [ 9542.149939] Lustre: Skipped 1 previous similar message [ 9544.573088] LustreError: 354709:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9544.582860] LustreError: 354709:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9544.759040] Lustre: server umount lustre-OST0000 complete [ 9549.794416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9549.804698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.356476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9552.363107] Lustre: Skipped 2 previous similar messages [ 9554.096414] LustreError: 354912:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9554.100064] LustreError: 354912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9554.233735] Lustre: server umount lustre-MDT0000 complete [ 9557.474903] LustreError: 351778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9557.493953] LustreError: 351778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9557.727482] LustreError: 353514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351348 with bad export cookie 5547999545161425542 [ 9557.728291] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.754402] LustreError: 353514:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9558.064822] Lustre: server umount lustre-MDT0001 complete [ 9565.629946] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:49:13 (1763351353) [ 9571.969367] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:49:19 (1763351359) [ 9578.158114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9578.451383] LustreError: 356128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9578.527340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.338968] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9583.587220] LustreError: 356128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9587.170530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9590.206740] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9594.909547] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9598.657204] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9605.397754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9611.795528] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9616.770184] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9629.716452] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9631.225562] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9632.891409] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9634.511069] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9636.266928] Lustre: Failing over lustre-MDT0000 [ 9636.432221] LustreError: 358589:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9636.435217] LustreError: 358589:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9638.369709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9638.374422] LustreError: 356123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9638.428034] Lustre: Skipped 3 previous similar messages [ 9638.450947] LustreError: 356123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9638.542452] Lustre: server umount lustre-MDT0000 complete [ 9643.496426] LustreError: 356122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9643.509660] LustreError: 356122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9653.728478] Lustre: 342863:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351428/real 1763351428] req@000000001e6d1680 x1849007636844928/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763351444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9653.740801] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9653.744564] LustreError: 356124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.824530] LustreError: 356124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9657.130219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.219684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.226955] Lustre: Skipped 2 previous similar messages [ 9664.283780] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9665.770634] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9667.206328] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9669.611355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9669.632783] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9669.697242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9673.320340] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9674.934480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9678.553349] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9680.120589] Lustre: Failing over lustre-MDT0000 [ 9680.302413] LustreError: 359670:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9680.306065] LustreError: 359670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9680.362311] Lustre: server umount lustre-MDT0000 complete [ 9681.173804] LustreError: 356124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.198525] LustreError: 356124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9684.963416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9684.965245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9684.977038] Lustre: Skipped 2 previous similar messages [ 9697.116629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9697.205318] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9697.438704] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9697.570227] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9700.725038] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9702.895125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9702.904882] Lustre: Skipped 2 previous similar messages [ 9702.928316] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9702.985407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9708.289943] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9710.198887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9716.194224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.196481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9716.204362] Lustre: Skipped 2 previous similar messages [ 9716.210968] Lustre: Skipped 1 previous similar message [ 9721.566638] LustreError: 360701:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9721.569646] LustreError: 360701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9721.655896] Lustre: server umount lustre-MDT0000 complete [ 9724.923285] LustreError: 356109:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351515 with bad export cookie 5547999545161428237 [ 9724.931598] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9724.938344] LustreError: 356109:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9725.247492] Lustre: server umount lustre-MDT0001 complete [ 9738.631363] Lustre: server umount lustre-OST0000 complete [ 9750.287288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.597622] LustreError: 361931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.610152] LustreError: 361931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 9750.692092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.700411] Lustre: Skipped 1 previous similar message [ 9754.454113] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9761.850665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9765.775858] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9770.359308] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9773.729462] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9779.001654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9783.301732] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9787.299983] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9795.643265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9802.721170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.726804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.750538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.762290] Lustre: Skipped 3 previous similar messages [ 9807.497358] LustreError: 364195:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9807.538244] LustreError: 364195:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9807.739529] Lustre: server umount lustre-OST0000 complete [ 9813.477279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9813.485040] LustreError: Skipped 1 previous similar message [ 9817.591138] Lustre: server umount lustre-MDT0000 complete [ 9820.328457] LustreError: 361911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351610 with bad export cookie 5547999545161428972 [ 9820.332986] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9820.333645] LustreError: 361911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9820.536288] Lustre: server umount lustre-MDT0001 complete [ 9827.463538] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9829.452821] Key type lgssc unregistered [ 9829.663461] LNet: 365131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9829.672117] LNetError: 365131:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9829.689886] LNet: Removed LNI 192.168.202.153@tcp [ 9830.371628] Key type .llcrypt unregistered [ 9830.373424] Key type ._llcrypt unregistered [ 9852.405407] Key type ._llcrypt registered [ 9852.407028] Key type .llcrypt registered [ 9852.492209] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 9853.163303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9853.214623] alg: No test for adler32 (adler32-zlib) [ 9854.159081] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 9854.339741] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 9855.977180] Key type lgssc registered [ 9856.717577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9863.435475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9863.463549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9865.035965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.471712] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9875.432225] LustreError: 366632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9875.441917] LustreError: 366632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9875.753954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9876.100591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9879.571205] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9884.278670] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9887.733808] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9893.976112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9894.333614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9900.037025] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9905.269304] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9909.693056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9919.970722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.988606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9919.998944] Lustre: Skipped 1 previous similar message [ 9925.094180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9925.097258] Lustre: Skipped 4 previous similar messages [ 9925.716598] LustreError: 368900:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9925.859022] Lustre: server umount lustre-MDT0000 complete [ 9929.346174] LustreError: 366612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351719 with bad export cookie 13646106165535674309 [ 9929.348404] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.350231] LustreError: 366612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9929.460268] LustreError: 369102:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9929.470380] LustreError: 369102:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9929.561349] Lustre: server umount lustre-MDT0001 complete [ 9943.457841] LustreError: 369304:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9943.460696] LustreError: 369304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9943.682657] Lustre: server umount lustre-OST0000 complete [ 9956.859474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.358152] LustreError: 370035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.453484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9960.896686] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9962.468941] LustreError: 370035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9967.595003] LustreError: 370035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.898888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9969.252761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9972.647500] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9977.263917] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9980.408510] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9986.260234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9986.527427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.868565] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 9995.559753] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10001.915706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10017.252692] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.257144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10017.276397] Lustre: Skipped 2 previous similar messages [10022.375280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10022.384337] Lustre: Skipped 4 previous similar messages [10023.025281] LustreError: 372403:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10023.029327] LustreError: 372403:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10023.122306] Lustre: server umount lustre-MDT0000 complete [10026.586632] LustreError: 370015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351816 with bad export cookie 13646106165535675429 [10026.589142] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10026.595528] LustreError: 370015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10026.938137] Lustre: server umount lustre-MDT0001 complete [10039.804407] LustreError: 372806:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10039.807223] LustreError: 372806:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10039.940250] Lustre: server umount lustre-OST0000 complete [10051.759212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.104253] LustreError: 373634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.200882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10055.257722] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10061.722500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10064.801916] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10069.332607] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10071.973196] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10077.422493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10078.774272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10083.049409] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10087.121865] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10092.007642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.007986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10092.022880] Lustre: Skipped 1 previous similar message [10092.028121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10092.043278] Lustre: Skipped 1 previous similar message [10095.986203] LustreError: 375878:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10095.989168] LustreError: 375878:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10096.056907] Lustre: server umount lustre-OST0000 complete [10100.194072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10100.197466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10100.201985] Lustre: Skipped 2 previous similar messages [10100.210203] Lustre: Skipped 3 previous similar messages [10105.221600] Lustre: server umount lustre-MDT0000 complete [10105.314996] LustreError: 374423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10105.325193] LustreError: 374423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10107.801888] LustreError: 373613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351898 with bad export cookie 13646106165535676444 [10107.806114] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10107.819792] LustreError: 373613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10108.033747] Lustre: server umount lustre-MDT0001 complete [10116.236193] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10118.672327] Key type lgssc unregistered [10118.909714] LNet: 376815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10118.915128] LNetError: 376815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10118.930981] LNet: Removed LNI 192.168.202.153@tcp [10119.767432] Key type .llcrypt unregistered [10119.772537] Key type ._llcrypt unregistered [10141.829154] Key type ._llcrypt registered [10141.830777] Key type .llcrypt registered [10141.924191] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10142.664232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10142.932725] alg: No test for adler32 (adler32-zlib) [10144.065625] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10144.348802] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10146.000188] Key type lgssc registered [10146.906807] Lustre: Echo OBD driver; http://www.lustre.org/ [10152.790830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10152.813678] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10152.820927] LustreError: 378234:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10152.837818] LustreError: 378234:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [10152.842830] LustreError: 378234:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10152.852567] LustreError: 378234:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10152.856440] LustreError: 378234:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10157.818320] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [10157.823243] LustreError: 378633:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10157.841984] LustreError: 378633:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [10157.845824] LustreError: 378633:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10157.850711] LustreError: 378633:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10157.857281] LustreError: 378633:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10169.709430] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10171.560307] Key type lgssc unregistered [10171.833772] LNet: 379618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10171.840516] LNetError: 379618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10171.852188] LNet: Removed LNI 192.168.202.153@tcp [10172.430493] Key type .llcrypt unregistered [10172.432206] Key type ._llcrypt unregistered [10184.541104] Key type ._llcrypt registered [10184.544821] Key type .llcrypt registered [10184.641403] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [10195.036803] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10195.697096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10195.712878] alg: No test for adler32 (adler32-zlib) [10196.730572] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10196.946718] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10198.600295] Key type lgssc registered [10199.518353] Lustre: Echo OBD driver; http://www.lustre.org/ [10205.796525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10210.303197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10214.986332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10220.069381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10224.884552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10224.932961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10224.949276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10226.090425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10226.113625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10226.166719] Lustre: lustre-MDT0000: new disk, initializing [10226.295225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10226.317782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10229.084303] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10236.574983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10236.611526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10236.669534] Lustre: 382728:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10236.718453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10236.722318] Lustre: Skipped 1 previous similar message [10236.790760] Lustre: lustre-MDT0001: new disk, initializing [10236.887195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10236.939417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10236.943731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10239.695888] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10245.265073] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10248.079988] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10252.324300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10252.378270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10252.538620] Lustre: lustre-OST0000: new disk, initializing [10252.545356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10252.600899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10256.609960] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10260.515971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10260.521177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10260.569826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10261.775930] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10264.131647] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10264.245926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10266.335416] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10266.437492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10270.691756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10270.697809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.707143] LustreError: Skipped 1 previous similar message [10270.714881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10270.720279] Lustre: Skipped 1 previous similar message [10273.529154] LustreError: 385668:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10273.598320] Lustre: server umount lustre-OST0000 complete [10277.859018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10277.864853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.880438] Lustre: Skipped 1 previous similar message [10277.884781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10280.931880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10280.931937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.049867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.053724] Lustre: Skipped 2 previous similar messages [10290.155446] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10290.241000] LustreError: 385880:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10290.245026] LustreError: 385880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10290.293374] Lustre: server umount lustre-MDT0000 complete [10292.467552] LustreError: 382722:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352082 with bad export cookie 10192400751877584266 [10292.470514] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10292.476216] LustreError: 382722:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10292.574230] LustreError: 386082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10292.577928] LustreError: 386082:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10292.700256] Lustre: server umount lustre-MDT0001 complete [10297.458599] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10298.344074] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:01:26 (1763352086) [10302.146149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.446869] LustreError: 386802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.456822] LustreError: 386802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10302.493748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.676320] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10307.557753] LustreError: 386802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.334660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10309.518772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10311.735706] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10314.933290] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10316.901364] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10320.943591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10321.149578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10324.867554] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10328.476146] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10333.337480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10333.383357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10333.504437] Lustre: lustre-OST0001: new disk, initializing [10333.508682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10336.818072] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10341.879716] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10342.956067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10342.962373] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10343.024627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10348.008492] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10348.009162] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.018921] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10348.029125] Lustre: Skipped 1 previous similar message [10348.038150] Lustre: Skipped 1 previous similar message [10350.465243] LustreError: 390001:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [10350.467924] LustreError: 390001:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10350.511756] Lustre: server umount lustre-OST0001 complete [10351.017586] LustreError: 388399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10351.024107] LustreError: 388399:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10356.198073] LustreError: 388401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.207043] LustreError: 388401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10361.313665] LustreError: 388401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10361.328039] LustreError: 388401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10365.410205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10365.416237] LustreError: Skipped 1 previous similar message [10365.418562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.429515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10365.432484] Lustre: Skipped 1 previous similar message [10370.296149] LustreError: 390227:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10370.299185] LustreError: 390227:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10370.386665] Lustre: server umount lustre-OST0000 complete [10376.164392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10376.170764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.176313] Lustre: Skipped 1 previous similar message [10378.653510] LustreError: 390429:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [10378.659053] LustreError: 390429:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10378.729615] Lustre: server umount lustre-MDT0000 complete [10381.401746] LustreError: 387521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352171 with bad export cookie 10192400751877585148 [10381.406351] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10381.412317] LustreError: 387521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10387.788407] Lustre: server umount lustre-MDT0001 complete [10393.710716] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10395.232146] Key type lgssc unregistered [10395.385926] LNet: 391162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10395.391160] LNetError: 391162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10396.454772] LNet: Removed LNI 192.168.202.153@tcp [10396.861778] Key type .llcrypt unregistered [10396.863204] Key type ._llcrypt unregistered [10417.865587] Key type ._llcrypt registered [10417.867204] Key type .llcrypt registered [10417.933917] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [10427.949303] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10428.356406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10428.434296] alg: No test for adler32 (adler32-zlib) [10429.353971] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10429.477275] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10431.080987] Key type lgssc registered [10431.639213] Lustre: Echo OBD driver; http://www.lustre.org/ [10436.352139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10440.216725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10443.918578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10447.594856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10451.354481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10451.388369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10451.398989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.516363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10452.536037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10452.597400] Lustre: lustre-MDT0000: new disk, initializing [10452.661757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10452.675121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10454.711368] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10460.626701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10460.667335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10460.723535] Lustre: 395085:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10460.758238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10460.761838] Lustre: Skipped 1 previous similar message [10460.805179] Lustre: lustre-MDT0001: new disk, initializing [10460.854515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10460.878719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10460.884472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10462.749345] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10467.182282] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10469.277673] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10473.058957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10473.099216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10473.235349] Lustre: lustre-OST0000: new disk, initializing [10473.238363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10473.299125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10476.302603] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10480.678516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10480.686962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10480.742920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10480.832569] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10482.808817] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10482.907662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10484.855113] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10484.949040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10490.849749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10490.850289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.852030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10490.854407] LustreError: Skipped 1 previous similar message [10490.859817] Lustre: Skipped 1 previous similar message [10495.975565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10495.978206] Lustre: Skipped 2 previous similar messages [10500.064136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10500.089824] LustreError: 398023:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10500.156064] Lustre: server umount lustre-OST0000 complete [10502.114878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10502.119400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.129400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.210489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.213711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.221964] Lustre: Skipped 1 previous similar message [10507.978032] LustreError: 398226:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10507.979752] LustreError: 398226:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10508.024254] Lustre: server umount lustre-MDT0000 complete [10509.933728] LustreError: 395077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352300 with bad export cookie 14433430469403139156 [10509.936981] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10509.939504] LustreError: 395077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10510.027546] LustreError: 398429:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10510.029795] LustreError: 398429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10510.114655] Lustre: server umount lustre-MDT0001 complete [10514.093336] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 23:05:02 (1763352302) [10518.467594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10522.371763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.592286] Lustre: lustre-MDT0000: new disk, initializing [10522.642850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10524.351194] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10527.715875] LustreError: 399696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.733262] LustreError: 399696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10528.243270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10528.457697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10530.279462] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10533.204800] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10535.367129] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10537.953356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.954229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10537.959627] Lustre: Skipped 1 previous similar message [10542.678114] LustreError: 401038:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10542.684183] LustreError: 401038:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10542.742949] Lustre: server umount lustre-MDT0000 complete [10544.101480] LustreError: 399690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.115596] LustreError: 399690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10544.754344] LustreError: 401040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352335 with bad export cookie 14433430469403140038 [10544.757374] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10544.758971] LustreError: 401040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10544.917666] Lustre: server umount lustre-MDT0001 complete [10549.430664] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:05:38 (1763352338) [10552.637989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.885050] LustreError: 401960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.923476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.970577] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10557.921149] LustreError: 401960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.420662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10561.430358] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10563.896369] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10565.622876] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10568.910520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10569.121653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10569.132387] Lustre: Skipped 1 previous similar message [10571.775894] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10574.410150] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10588.158555] Lustre: Permanently deactivating lustre-OST0000 [10588.167214] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10597.883312] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10598.910568] LustreError: 404767:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10598.915404] LustreError: 404767:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10598.918730] LustreError: 404767:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10599.943281] LustreError: 404815:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10599.946543] LustreError: 404815:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10606.904091] LustreError: 404912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10606.911133] LustreError: 404912:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10606.990584] Lustre: server umount lustre-OST0000 complete [10608.441145] LustreError: 405065:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10608.445814] LustreError: 405065:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10610.146182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.147753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.150368] Lustre: Skipped 1 previous similar message [10610.154796] Lustre: Skipped 4 previous similar messages [10619.375479] LustreError: 405162:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10619.379261] LustreError: 405162:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10619.419121] Lustre: server umount lustre-MDT0000 complete [10620.386523] LustreError: 401959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.393167] LustreError: 401959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10621.157313] LustreError: 404328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352411 with bad export cookie 14433430469403140577 [10621.160238] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10621.162382] LustreError: 404328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10630.711589] Lustre: server umount lustre-MDT0001 complete [10635.015804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.920402] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10638.326669] LustreError: 406207:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10641.487458] Lustre: 406545:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.153@tcp [10642.528154] LustreError: 406642:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10642.531102] LustreError: 406642:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10642.562277] Lustre: server umount MGS complete [10648.232612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.440721] LustreError: 407270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.448307] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10648.452312] Lustre: Skipped 1 previous similar message [10648.476714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10650.358600] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10654.438744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10654.592945] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10656.456604] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10658.816588] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10660.337433] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10661.131153] Lustre: Permanently reactivating lustre-OST0000 [10661.138236] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10661.140866] Lustre: Skipped 2 previous similar messages [10681.312258] LustreError: 408626:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10684.819822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10687.483364] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10689.816413] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10706.401596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10706.405516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.406562] LustreError: Skipped 1 previous similar message [10706.407270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10706.411381] Lustre: Skipped 1 previous similar message [10706.413219] Lustre: Skipped 2 previous similar messages [10708.485305] LustreError: 410418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10708.488470] LustreError: 410418:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10708.538379] Lustre: server umount lustre-OST0000 complete [10711.011749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10716.641877] LustreError: 407270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.650843] LustreError: 407270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10716.652920] Lustre: server umount lustre-MDT0000 complete [10718.283490] LustreError: 407251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352508 with bad export cookie 14433430469403141529 [10718.286805] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10718.288574] LustreError: 407251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10718.427985] Lustre: server umount lustre-MDT0001 complete [10723.139653] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10724.461752] Key type lgssc unregistered [10724.619796] LNet: 411352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10724.623588] LNetError: 411352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10724.630364] LNet: Removed LNI 192.168.202.153@tcp [10725.000808] Key type .llcrypt unregistered [10725.002574] Key type ._llcrypt unregistered [10733.880135] Key type ._llcrypt registered [10733.881638] Key type .llcrypt registered [10733.928691] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [10741.459488] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10741.815805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10741.824097] alg: No test for adler32 (adler32-zlib) [10742.705534] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10742.808365] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10744.400232] Key type lgssc registered [10744.865538] Lustre: Echo OBD driver; http://www.lustre.org/ [10749.158102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10751.935281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10754.592712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10757.319787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10760.332762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10760.362430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10760.372298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.483250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10761.503032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10761.544304] Lustre: lustre-MDT0000: new disk, initializing [10761.598094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10761.606891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10763.213169] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10768.050088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10768.081363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10768.128054] Lustre: 414472:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10768.161481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10768.164119] Lustre: Skipped 1 previous similar message [10768.205939] Lustre: lustre-MDT0001: new disk, initializing [10768.252224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10768.268263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10768.272128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10769.917784] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10773.673187] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10775.371963] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10778.530066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10778.570653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10778.701209] Lustre: lustre-OST0000: new disk, initializing [10778.704511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10778.752436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10781.385606] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10784.780052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10784.785563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10784.838169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10785.283169] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10787.039264] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10787.130311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10788.786326] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10788.879878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10789.858430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10789.859121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10789.864790] LustreError: Skipped 1 previous similar message [10789.866147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10789.870817] Lustre: Skipped 1 previous similar message [10791.796791] LustreError: 417409:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10791.872379] Lustre: server umount lustre-OST0000 complete [10793.953057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10793.956262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10793.962781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10793.965436] Lustre: Skipped 1 previous similar message [10794.977497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10794.977847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10794.987292] Lustre: Skipped 1 previous similar message [10799.625670] LustreError: 417609:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10799.628128] LustreError: 417609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10799.664794] Lustre: server umount lustre-MDT0000 complete [10801.340603] LustreError: 414464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352591 with bad export cookie 10696908784637877946 [10801.343599] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10801.345050] LustreError: 414464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10801.411237] LustreError: 417812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10801.413290] LustreError: 417812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10801.479314] Lustre: server umount lustre-MDT0001 complete [10805.006839] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10805.736846] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:09:54 (1763352594) [10808.758767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10808.981163] LustreError: 418531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.989879] LustreError: 418531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10809.022117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10810.840990] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10814.434422] LustreError: 418530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.441946] LustreError: 418530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10814.943077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10815.127516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10816.936179] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10819.415683] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10821.030177] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10824.093947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10824.280705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10826.828602] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10829.230241] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10830.173573] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10837.327174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10840.037302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10840.037376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.038429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10840.050775] Lustre: Skipped 1 previous similar message [10845.155691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10845.158629] Lustre: Skipped 1 previous similar message [10845.301795] LustreError: 420968:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10845.305252] LustreError: 420968:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10845.364543] Lustre: server umount lustre-OST0000 complete [10850.274776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.204232] LustreError: 421169:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10853.206773] LustreError: 421169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10853.268105] Lustre: server umount lustre-MDT0000 complete [10855.376311] LustreError: 418512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352645 with bad export cookie 10696908784637878828 [10855.381570] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10855.383117] LustreError: 418512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10855.549772] Lustre: server umount lustre-MDT0001 complete [10860.612818] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10862.102713] Key type lgssc unregistered [10862.257823] LNet: 421903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10862.261702] LNetError: 421903:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10862.271684] LNet: Removed LNI 192.168.202.153@tcp [10862.711535] Key type .llcrypt unregistered [10862.712977] Key type ._llcrypt unregistered [10873.519270] Key type ._llcrypt registered [10873.521354] Key type .llcrypt registered [10873.584266] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10873.765440] Key type .llcrypt unregistered [10873.766890] Key type ._llcrypt unregistered [10882.648808] Key type ._llcrypt registered [10882.650281] Key type .llcrypt registered [10882.706699] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10883.245529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10883.261824] alg: No test for adler32 (adler32-zlib) [10884.153230] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10884.267206] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10885.864284] Key type lgssc registered [10886.378739] Lustre: Echo OBD driver; http://www.lustre.org/ [10890.097191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10890.109324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10891.365138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10893.057228] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10896.640903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10896.794671] LustreError: 424048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.801476] LustreError: 424048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10896.803961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10899.276217] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10900.898253] LustreError: 424052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10900.907273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10900.908538] LustreError: 424052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10901.756687] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10905.110933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10905.319208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10907.188079] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [10915.811910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10915.812757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10915.813354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10915.817540] Lustre: Skipped 1 previous similar message [10920.931603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10920.934520] Lustre: Skipped 1 previous similar message [10921.519750] LustreError: 426105:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10921.570698] Lustre: server umount lustre-OST0000 complete [10926.050036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10926.050129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10926.053740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10926.053748] Lustre: Skipped 1 previous similar message [10926.070537] Lustre: Skipped 1 previous similar message [10929.285037] LustreError: 426306:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10929.289018] LustreError: 426306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10929.332154] Lustre: server umount lustre-MDT0000 complete [10931.169298] LustreError: 425269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10931.175599] LustreError: 425269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10931.270743] LustreError: 424033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352721 with bad export cookie 1116374279898539214 [10931.273601] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10931.275885] LustreError: 424033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10931.364709] LustreError: 426509:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10931.367492] LustreError: 426509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10931.447091] Lustre: server umount lustre-MDT0001 complete [10936.232068] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [10937.610964] Key type lgssc unregistered [10937.779524] LNet: 427041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10937.782721] LNetError: 427041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10937.789957] LNet: Removed LNI 192.168.202.153@tcp [10938.241924] Key type .llcrypt unregistered [10938.244275] Key type ._llcrypt unregistered [10950.066418] Key type ._llcrypt registered [10950.067983] Key type .llcrypt registered [10950.122274] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [10950.480486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10950.487335] alg: No test for adler32 (adler32-zlib) [10951.382989] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [10951.496762] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [10953.096263] Key type lgssc registered [10953.575681] Lustre: Echo OBD driver; http://www.lustre.org/ [10957.043158] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10957.053611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10972.512547] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10977.632348] LustreError: 428388:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [11004.256760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11006.825913] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11009.383691] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11012.377070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.642117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11012.686942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11014.320678] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11017.698160] LustreError: 428415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.704869] LustreError: 428415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11018.001539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11018.220678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11020.181620] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11022.699514] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11024.153221] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11027.936776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11027.939488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11027.945726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11028.449244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11028.452484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11028.457934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11029.479510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11033.133148] LustreError: 430668:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [11033.184529] Lustre: server umount lustre-OST0000 complete [11037.155776] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11037.159198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11037.166167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11037.168734] Lustre: Skipped 1 previous similar message [11039.714245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.024300] LustreError: 430870:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11041.026194] LustreError: 430870:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11041.058189] Lustre: server umount lustre-MDT0000 complete [11042.722293] LustreError: 430582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352833 with bad export cookie 10258833840740278841 [11042.725165] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11042.727702] LustreError: 430582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11042.814323] LustreError: 431073:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11042.817266] LustreError: 431073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11042.899518] Lustre: server umount lustre-MDT0001 complete [11047.361497] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11048.622615] Key type lgssc unregistered [11048.758448] LNet: 431604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11048.761682] LNetError: 431604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11048.768348] LNet: Removed LNI 192.168.202.153@tcp [11049.135471] Key type .llcrypt unregistered [11049.137266] Key type ._llcrypt unregistered [11060.527276] Key type ._llcrypt registered [11060.528867] Key type .llcrypt registered [11060.575963] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11060.997051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11061.038071] alg: No test for adler32 (adler32-zlib) [11061.939671] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11062.042206] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11063.640217] Key type lgssc registered [11064.116607] Lustre: Echo OBD driver; http://www.lustre.org/ [11067.395367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11067.408681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.623162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11070.342490] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11074.015339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11074.184886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11075.807908] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11078.066059] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11079.597453] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11082.544076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11082.695669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11085.200653] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11087.472298] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11092.970372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11100.132077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11100.135433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11100.142738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11102.951918] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [11103.202652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.023857] LustreError: 435243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11105.062756] Lustre: server umount lustre-MDT0000 complete [11106.000813] LustreError: 432961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.53@tcp arrived at 1763352896 with bad export cookie 9713973151216062999 [11106.007013] LustreError: 432961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11106.814249] LustreError: 432961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352897 with bad export cookie 9713973151216062348 [11106.818688] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11106.820935] LustreError: 432961:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11108.321139] LustreError: 434139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.321661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11108.326675] LustreError: 434139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11108.336130] Lustre: Skipped 1 previous similar message [11108.338847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11108.341387] Lustre: Skipped 3 previous similar messages [11111.329481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.329487] LustreError: 434139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.338872] LustreError: 434139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11112.879139] LustreError: 435447:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11112.880972] LustreError: 435447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11112.926422] Lustre: server umount lustre-OST0000 complete [11115.559583] LustreError: 435801:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11115.562199] LustreError: 435801:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11115.622061] Lustre: server umount lustre-MDT0001 complete [11119.790425] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11121.099296] Key type lgssc unregistered [11121.252681] LNet: 436332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11121.256593] LNetError: 436332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11121.268410] LNet: Removed LNI 192.168.202.153@tcp [11121.653798] Key type .llcrypt unregistered [11121.655764] Key type ._llcrypt unregistered [11133.318231] Key type ._llcrypt registered [11133.319751] Key type .llcrypt registered [11133.367973] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11133.743027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11133.780096] alg: No test for adler32 (adler32-zlib) [11134.681120] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11134.777366] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11136.368228] Key type lgssc registered [11136.884724] Lustre: Echo OBD driver; http://www.lustre.org/ [11140.537059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11140.546560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.776340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11143.455138] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11147.630486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11147.821654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11149.582511] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11152.210071] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11153.909664] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11156.963800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11157.132842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11159.693431] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11161.838806] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11167.401089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11172.321339] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11172.325228] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11172.333809] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11172.838081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11172.848089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11172.850870] Lustre: Skipped 1 previous similar message [11174.418749] LustreError: 439952:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11174.473239] Lustre: server umount lustre-MDT0001 complete [11177.954798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11177.955643] LustreError: 437706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11177.958596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11177.959719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11177.965138] LustreError: 437706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11182.971094] LustreError: 440154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11182.973025] LustreError: 440154:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11183.034300] Lustre: server umount lustre-OST0000 complete [11183.074821] LustreError: 437706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.599558] LustreError: 440356:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11184.601599] LustreError: 440356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11184.670278] Lustre: server umount lustre-MDT0000 complete [11190.117851] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11191.363978] Key type lgssc unregistered [11191.516528] LNet: 441041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11191.520274] LNetError: 441041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11191.526397] LNet: Removed LNI 192.168.202.153@tcp [11191.869652] Key type .llcrypt unregistered [11191.871280] Key type ._llcrypt unregistered [11202.888570] Key type ._llcrypt registered [11202.890015] Key type .llcrypt registered [11202.939129] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [11210.092523] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11210.491604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11210.542365] alg: No test for adler32 (adler32-zlib) [11211.427821] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11211.533339] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11213.128289] Key type lgssc registered [11213.614945] Lustre: Echo OBD driver; http://www.lustre.org/ [11217.340254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11220.201610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11222.870332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11225.743545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11229.638626] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11233.472889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11236.503167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11236.530400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11236.538332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11237.655949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11237.677462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11237.721620] Lustre: lustre-MDT0000: new disk, initializing [11237.760948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11237.770667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11239.384624] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11243.497750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11246.448713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11246.480073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11246.604375] Lustre: lustre-OST0000: new disk, initializing [11246.607136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11246.609461] Lustre: Skipped 1 previous similar message [11246.638673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11249.009836] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11252.721303] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11254.793848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11254.798717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11254.813094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11262.799786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11265.603795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11265.636688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11265.682906] Lustre: 444988:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11265.716145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11265.756635] Lustre: lustre-MDT0001: new disk, initializing [11265.811532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11265.830827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11265.836438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11267.396453] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11271.600513] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11271.687452] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11273.323931] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11273.429094] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11277.549205] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [11277.794893] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11277.795493] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.804138] Lustre: Skipped 1 previous similar message [11281.667822] LustreError: 448407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11281.726208] Lustre: server umount lustre-MDT0001 complete [11288.035456] LustreError: 444995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.043300] LustreError: 444995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11293.154418] LustreError: 444995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.160769] LustreError: 444995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11295.713061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11295.716328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11295.721216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11295.723981] Lustre: Skipped 2 previous similar messages [11298.274072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11298.274947] LustreError: 448030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.725079] LustreError: 448680:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11300.727889] LustreError: 448680:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11300.782686] Lustre: server umount lustre-OST0000 complete [11308.481194] LustreError: 448881:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11308.483517] LustreError: 448881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11308.550253] Lustre: server umount lustre-MDT0000 complete [11314.016693] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11315.243400] Key type lgssc unregistered [11315.383577] LNet: 449566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11315.387633] LNetError: 449566:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11315.395391] LNet: Removed LNI 192.168.202.153@tcp [11315.774908] Key type .llcrypt unregistered [11315.776562] Key type ._llcrypt unregistered [11331.995893] Key type ._llcrypt registered [11331.997580] Key type .llcrypt registered [11332.045259] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11332.177595] Key type .llcrypt unregistered [11332.179066] Key type ._llcrypt unregistered [11339.722205] Key type ._llcrypt registered [11339.723743] Key type .llcrypt registered [11339.768913] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11340.115324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11340.123913] alg: No test for adler32 (adler32-zlib) [11341.003886] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11341.105138] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11342.696215] Key type lgssc registered [11343.174210] Lustre: Echo OBD driver; http://www.lustre.org/ [11346.348146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11348.943315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11351.732643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11354.636146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11354.661346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11354.673197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11355.779533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11355.795483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11355.837493] Lustre: lustre-MDT0000: new disk, initializing [11355.879529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11355.889430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11357.538655] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11362.430817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11362.460956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11362.505338] Lustre: 452895:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11362.540895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11362.543982] Lustre: Skipped 1 previous similar message [11362.584727] Lustre: lustre-MDT0001: new disk, initializing [11362.629527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11362.647257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11362.651700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11364.238182] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11367.733711] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11369.226250] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11372.202035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11372.230678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11372.359338] Lustre: lustre-OST0000: new disk, initializing [11372.362132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11372.402414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11373.569776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11373.574145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11373.630856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11374.729890] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11378.369403] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11383.264860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11383.268556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11383.274674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11383.780954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11383.787538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11383.790020] Lustre: Skipped 1 previous similar message [11387.083943] LustreError: 455338:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11387.131163] Lustre: server umount lustre-MDT0000 complete [11388.705282] LustreError: 454643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353179 with bad export cookie 1876636005456618069 [11388.707928] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11388.710033] LustreError: 454643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11388.780186] LustreError: 455540:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11388.782704] LustreError: 455540:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11388.851488] Lustre: server umount lustre-MDT0001 complete [11400.632087] LustreError: 455742:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11400.634766] LustreError: 455742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11400.691986] Lustre: server umount lustre-OST0000 complete [11404.893967] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:19:53 (1763353193) [11405.354311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11408.272159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.503053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11410.147766] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11413.986377] LustreError: 456588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11413.989426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11413.995393] LustreError: 456588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11414.185449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11415.892874] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11418.119264] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11419.573235] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11422.376522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11422.411940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11422.492298] Lustre: Found index 0 for lustre-OST0000, updating log [11422.571395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11424.829375] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11427.219867] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11432.808068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11440.097275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11440.099587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11440.103397] Lustre: Skipped 1 previous similar message [11440.105441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11445.294093] LustreError: 458847:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11445.297089] LustreError: 458847:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11445.336907] Lustre: server umount lustre-OST0000 complete [11446.425045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11447.157437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.158129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.162115] Lustre: Skipped 2 previous similar messages [11447.164775] Lustre: Skipped 3 previous similar messages [11453.313640] LustreError: 459102:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11453.316973] LustreError: 459102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11453.350043] Lustre: server umount lustre-MDT0000 complete [11453.409394] LustreError: 457729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.419046] LustreError: 457729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11454.915544] LustreError: 459104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353245 with bad export cookie 1876636005456619448 [11454.918069] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11454.919812] LustreError: 459104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11455.054562] Lustre: server umount lustre-MDT0001 complete [11458.622903] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:20:47 (1763353247) [11459.038915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11459.442903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11459.887978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11460.314576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11460.718589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11461.115111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11461.543246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11461.928742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11462.316937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11462.706987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11463.076732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11463.459727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11463.907783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11464.358656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11464.803851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11465.217921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11465.610199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11465.998415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11466.370959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11466.776228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11467.170292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11467.554061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11467.922080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11468.311211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11468.701332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11469.071162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11469.461080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11469.880477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11470.306439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11470.764213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11471.211992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11471.673325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11472.153680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11472.569743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11472.961413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11473.341483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11473.732302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11474.145719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11474.541040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11474.928541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11475.300372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11475.701961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11476.071889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11476.436914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11476.813239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11477.230308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11477.281242] LNet: 1 local NIs in recovery (showing 1): 192.168.202.153@tcp [11477.606628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11477.993830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11478.407369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11478.795819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11479.172621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11482.153512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.370892] LustreError: 462686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.410377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11483.991350] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11487.713639] LustreError: 462686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11487.899897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11489.822988] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11492.088316] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11493.540630] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11496.395383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11496.437638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11496.550942] Lustre: Found index 0 for lustre-OST0000, updating log [11499.066667] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11501.344878] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11504.098329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11504.099557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11504.103747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11504.108904] Lustre: Skipped 2 previous similar messages [11508.274467] LustreError: 464971:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11508.276654] LustreError: 464971:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11508.324292] Lustre: server umount lustre-OST0000 complete [11512.993178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11512.995280] Lustre: Skipped 6 previous similar messages [11516.066096] LustreError: 462686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.071591] Lustre: server umount lustre-MDT0000 complete [11516.072073] LustreError: 462686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11517.594383] LustreError: 462666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353307 with bad export cookie 1876636005456620519 [11517.597436] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11517.599256] LustreError: 462666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11517.730627] Lustre: server umount lustre-MDT0001 complete [11518.850798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11525.343508] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [11528.995915] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11533.338019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11536.446635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11539.153573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11541.793203] LNet: 1 local NIs in recovery (showing 1): 192.168.202.153@tcp [11541.841371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11544.670806] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:22:13 (1763353333) [11561.974421] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11563.253823] Key type lgssc unregistered [11563.408924] LNet: 468955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11563.412878] LNetError: 468955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11687.337513] LNet: Removed LNI 192.168.202.153@tcp [11687.698821] Key type .llcrypt unregistered [11687.700465] Key type ._llcrypt unregistered [11695.366501] Key type ._llcrypt registered [11695.368111] Key type .llcrypt registered [11695.412883] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11695.783792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11695.796373] alg: No test for adler32 (adler32-zlib) [11696.700542] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11696.804876] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11698.400310] Key type lgssc registered [11698.829492] Lustre: Echo OBD driver; http://www.lustre.org/ [11700.520994] LNet: Added LNI 192.168.202.153@tcp100 [8/256/0/180] [11700.871399] LNet: Added LNI 192.168.202.153@tcp101 [8/256/0/180] [11701.232308] LNet: Added LNI 192.168.202.153@tcp102 [8/256/0/180] [11701.591138] LNet: Added LNI 192.168.202.153@tcp103 [8/256/0/180] [11701.956477] LNet: Added LNI 192.168.202.153@tcp104 [8/256/0/180] [11702.313091] LNet: Added LNI 192.168.202.153@tcp105 [8/256/0/180] [11702.681161] LNet: Added LNI 192.168.202.153@tcp106 [8/256/0/180] [11703.044747] LNet: Added LNI 192.168.202.153@tcp107 [8/256/0/180] [11703.376505] LNet: Added LNI 192.168.202.153@tcp108 [8/256/0/180] [11703.708597] LNet: Added LNI 192.168.202.153@tcp109 [8/256/0/180] [11704.061457] LNet: Added LNI 192.168.202.153@tcp110 [8/256/0/180] [11704.390564] LNet: Added LNI 192.168.202.153@tcp111 [8/256/0/180] [11704.721194] LNet: Added LNI 192.168.202.153@tcp112 [8/256/0/180] [11705.068798] LNet: Added LNI 192.168.202.153@tcp113 [8/256/0/180] [11705.417796] LNet: Added LNI 192.168.202.153@tcp114 [8/256/0/180] [11705.777506] LNet: Added LNI 192.168.202.153@tcp115 [8/256/0/180] [11706.128804] LNet: Added LNI 192.168.202.153@tcp116 [8/256/0/180] [11706.480972] LNet: Added LNI 192.168.202.153@tcp117 [8/256/0/180] [11706.822574] LNet: Added LNI 192.168.202.153@tcp118 [8/256/0/180] [11707.170698] LNet: Added LNI 192.168.202.153@tcp119 [8/256/0/180] [11707.506799] LNet: Added LNI 192.168.202.153@tcp120 [8/256/0/180] [11707.885588] LNet: Added LNI 192.168.202.153@tcp121 [8/256/0/180] [11708.247292] LNet: Added LNI 192.168.202.153@tcp122 [8/256/0/180] [11708.605856] LNet: Added LNI 192.168.202.153@tcp123 [8/256/0/180] [11708.973838] LNet: Added LNI 192.168.202.153@tcp124 [8/256/0/180] [11709.338443] LNet: Added LNI 192.168.202.153@tcp125 [8/256/0/180] [11709.719569] LNet: Added LNI 192.168.202.153@tcp126 [8/256/0/180] [11710.093293] LNet: Added LNI 192.168.202.153@tcp127 [8/256/0/180] [11710.456288] LNet: Added LNI 192.168.202.153@tcp128 [8/256/0/180] [11710.840299] LNet: Added LNI 192.168.202.153@tcp129 [8/256/0/180] [11711.218684] LNet: Added LNI 192.168.202.153@tcp130 [8/256/0/180] [11711.593950] LNet: Added LNI 192.168.202.153@tcp131 [8/256/0/180] [11711.974872] LNet: Added LNI 192.168.202.153@tcp132 [8/256/0/180] [11712.410346] LNet: Added LNI 192.168.202.153@tcp133 [8/256/0/180] [11712.805578] LNet: Added LNI 192.168.202.153@tcp134 [8/256/0/180] [11713.214023] LNet: Added LNI 192.168.202.153@tcp135 [8/256/0/180] [11716.078621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11716.104624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11716.110988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11717.161060] Lustre: 471958:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.153@tcp: can't add NID 192.168.202.153@tcp130: rc = -75 [11717.188169] LNetError: 471981:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11717.192928] LNetError: Protocol error connecting to 192.168.202.153@tcp115 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11717.196694] LNetError: Skipped 2 previous similar messages [11717.248983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11717.265098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11717.280156] LNet: 1 local NIs in recovery (showing 1): 192.168.202.153@tcp115 [11717.304953] Lustre: lustre-MDT0000: new disk, initializing [11717.353607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11717.362334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11718.242177] LNetError: 471988:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11718.246159] LNetError: 471988:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [11718.249477] LNetError: Protocol error connecting to 192.168.202.153@tcp101 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11718.899328] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11720.353382] LNetError: 471979:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11720.358149] LNetError: 471979:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [11720.362021] LNetError: Protocol error connecting to 192.168.202.153@tcp115 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11720.366983] LNetError: Skipped 1 previous similar message [11723.551814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11723.577068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11723.636137] Lustre: 471977:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11723.667255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11723.669717] Lustre: Skipped 1 previous similar message [11723.709518] Lustre: lustre-MDT0001: new disk, initializing [11723.723213] Lustre: 472761:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.153@tcp: can't add NID 192.168.202.153@tcp130: rc = -75 [11723.727104] Lustre: 472761:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11723.777968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11723.791635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11723.795699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11724.449211] LNetError: 471980:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11724.453000] LNetError: Protocol error connecting to 192.168.202.153@tcp115 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11725.288966] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11725.495220] Lustre: 473027:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.153@tcp: can't add NID 192.168.202.153@tcp130: rc = -75 [11725.501358] Lustre: 473027:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11728.760498] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11730.132181] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11732.642607] LNetError: 469421:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11732.647366] LNetError: Protocol error connecting to 192.168.202.153@tcp115 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11732.686352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11732.716158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11732.839883] Lustre: lustre-OST0000: new disk, initializing [11732.842944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11732.884582] Lustre: 473732:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.153@tcp: can't add NID 192.168.202.153@tcp130: rc = -75 [11732.887633] Lustre: 473732:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11732.909444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11733.432307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11733.436861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11733.499966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11734.985293] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11738.345633] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11743.717525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11743.717725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11743.720046] LustreError: Skipped 1 previous similar message [11743.726818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11743.730292] Lustre: Skipped 1 previous similar message [11747.057245] LustreError: 474566:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11747.108052] Lustre: server umount lustre-OST0000 complete [11748.833479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11748.834180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11748.839212] Lustre: Skipped 2 previous similar messages [11749.026253] LNetError: 469418:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11749.030166] LNetError: Protocol error connecting to 192.168.202.153@tcp115 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11753.955820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11753.959108] Lustre: Skipped 2 previous similar messages [11754.758593] LustreError: 474769:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11754.761325] LustreError: 474769:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11754.795334] Lustre: server umount lustre-MDT0000 complete [11756.294780] LustreError: 472760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353546 with bad export cookie 1726933963184742441 [11756.298178] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11756.299456] LustreError: 472760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11756.373241] LustreError: 474972:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11756.375648] LustreError: 474972:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11756.431750] Lustre: server umount lustre-MDT0001 complete [11760.595851] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11761.753580] Key type lgssc unregistered [11761.942120] LNet: 475511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11761.945520] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11761.948212] LNet: Removed LNI 192.168.202.153@tcp [11762.976546] LNet: Removed LNI 192.168.202.153@tcp100 [11762.978729] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11762.985084] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11762.989063] LNet: Removed LNI 192.168.202.153@tcp101 [11764.000153] LNet: Removed LNI 192.168.202.153@tcp102 [11764.002343] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11764.005961] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11764.008905] LNet: Removed LNI 192.168.202.153@tcp103 [11765.024209] LNet: Removed LNI 192.168.202.153@tcp104 [11766.048142] LNet: Removed LNI 192.168.202.153@tcp105 [11766.049746] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11766.052646] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11766.056286] LNet: Removed LNI 192.168.202.153@tcp106 [11767.072114] LNet: Removed LNI 192.168.202.153@tcp107 [11768.096133] LNet: Removed LNI 192.168.202.153@tcp108 [11769.120576] LNet: Removed LNI 192.168.202.153@tcp109 [11770.144145] LNet: Removed LNI 192.168.202.153@tcp110 [11770.145912] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11770.149192] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11770.152079] LNet: Removed LNI 192.168.202.153@tcp111 [11771.168155] LNet: Removed LNI 192.168.202.153@tcp112 [11772.192160] LNet: Removed LNI 192.168.202.153@tcp113 [11773.216307] LNet: Removed LNI 192.168.202.153@tcp114 [11773.218420] LNet: Removed LNI 192.168.202.153@tcp115 [11774.240216] LNet: Removed LNI 192.168.202.153@tcp116 [11775.264130] LNet: Removed LNI 192.168.202.153@tcp117 [11776.288251] LNet: Removed LNI 192.168.202.153@tcp118 [11777.312139] LNet: Removed LNI 192.168.202.153@tcp119 [11778.336206] LNet: Removed LNI 192.168.202.153@tcp120 [11778.339044] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11778.342697] LNetError: 475511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [11778.346244] LNet: Removed LNI 192.168.202.153@tcp121 [11778.348223] LNet: Removed LNI 192.168.202.153@tcp122 [11778.349761] LNet: Removed LNI 192.168.202.153@tcp123 [11778.350945] LNet: Removed LNI 192.168.202.153@tcp124 [11778.352837] LNet: Removed LNI 192.168.202.153@tcp125 [11778.354667] LNet: Removed LNI 192.168.202.153@tcp126 [11778.356043] LNet: Removed LNI 192.168.202.153@tcp127 [11778.357259] LNet: Removed LNI 192.168.202.153@tcp128 [11778.358502] LNet: Removed LNI 192.168.202.153@tcp129 [11778.360037] LNet: Removed LNI 192.168.202.153@tcp130 [11778.362099] LNet: Removed LNI 192.168.202.153@tcp131 [11778.364047] LNet: Removed LNI 192.168.202.153@tcp132 [11778.365376] LNet: Removed LNI 192.168.202.153@tcp133 [11778.367096] LNet: Removed LNI 192.168.202.153@tcp134 [11778.374542] LNet: Removed LNI 192.168.202.153@tcp135 [11778.698907] Key type .llcrypt unregistered [11778.700427] Key type ._llcrypt unregistered [11787.612958] Key type ._llcrypt registered [11787.614658] Key type .llcrypt registered [11787.662473] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [11794.321324] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11794.616200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11794.623449] alg: No test for adler32 (adler32-zlib) [11795.497103] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11795.590837] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11797.168182] Key type lgssc registered [11797.546492] Lustre: Echo OBD driver; http://www.lustre.org/ [11800.717539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11803.200050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11805.647320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11808.315474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11811.167090] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:26:39 (1763353599) [11818.120632] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11819.414325] Key type lgssc unregistered [11819.562729] LNet: 479527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11819.565962] LNetError: 479527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11819.578429] LNet: Removed LNI 192.168.202.153@tcp [11819.932790] Key type .llcrypt unregistered [11819.934114] Key type ._llcrypt unregistered [11826.875086] Key type ._llcrypt registered [11826.876036] Key type .llcrypt registered [11826.920332] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [11827.346380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11827.355255] alg: No test for adler32 (adler32-zlib) [11828.216554] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [11828.305567] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [11829.896145] Key type lgssc registered [11830.306324] Lustre: Echo OBD driver; http://www.lustre.org/ [11833.796895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11833.819267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11833.825580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11834.907606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11834.918171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11834.957095] Lustre: lustre-MDT0000: new disk, initializing [11834.984649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11834.991934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11836.312455] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11840.421221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11840.443398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11840.476518] Lustre: 480699:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11840.497123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11840.498625] Lustre: Skipped 1 previous similar message [11840.530526] Lustre: lustre-MDT0001: new disk, initializing [11840.560989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11840.572296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11840.576556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11841.848691] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11845.023490] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11846.308572] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11848.680183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11848.706041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11848.799243] Lustre: lustre-OST0000: new disk, initializing [11848.800782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11848.827725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11850.713882] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [11851.319084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11851.322100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11851.359501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11853.950759] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11855.157872] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing lnet_if_list [11856.766973] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11856.777606] LNet: Added LNI 192.168.202.153@tcp10 [8/256/0/180] [11858.622868] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11858.635916] LNet: Added LNI 192.168.202.153@tcp11 [8/256/0/180] [11859.981706] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11859.996584] LNet: Added LNI 192.168.202.153@tcp12 [8/256/0/180] [11861.444378] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11861.457485] LNet: Added LNI 192.168.202.153@tcp13 [8/256/0/180] [11862.869234] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11862.883324] LNet: Added LNI 192.168.202.153@tcp14 [8/256/0/180] [11864.342540] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11864.356449] LNet: Added LNI 192.168.202.153@tcp15 [8/256/0/180] [11865.877308] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11865.890551] LNet: Added LNI 192.168.202.153@tcp16 [8/256/0/180] [11867.451101] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11867.462875] LNet: Added LNI 192.168.202.153@tcp17 [8/256/0/180] [11869.042117] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11869.054830] LNet: Added LNI 192.168.202.153@tcp18 [8/256/0/180] [11870.623969] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11870.636311] LNet: Added LNI 192.168.202.153@tcp19 [8/256/0/180] [11872.298092] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11872.310676] LNet: Added LNI 192.168.202.153@tcp20 [8/256/0/180] [11874.111962] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11874.124089] LNet: Added LNI 192.168.202.153@tcp21 [8/256/0/180] [11875.861034] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11875.871270] LNet: Added LNI 192.168.202.153@tcp22 [8/256/0/180] [11877.545186] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11877.557525] LNet: Added LNI 192.168.202.153@tcp23 [8/256/0/180] [11879.390348] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11879.401141] LNet: Added LNI 192.168.202.153@tcp24 [8/256/0/180] [11881.161018] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11881.174863] LNet: Added LNI 192.168.202.153@tcp25 [8/256/0/180] [11882.988812] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11882.999798] LNet: Added LNI 192.168.202.153@tcp26 [8/256/0/180] [11884.883429] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11884.894863] LNet: Added LNI 192.168.202.153@tcp27 [8/256/0/180] [11884.900566] LNetError: 479960:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11884.905400] LNetError: Protocol error connecting to 192.168.202.153@tcp26 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11886.887287] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11886.898540] LNet: Added LNI 192.168.202.153@tcp28 [8/256/0/180] [11886.902027] LNetError: 479959:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11886.907945] LNetError: Protocol error connecting to 192.168.202.153@tcp26 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11888.813848] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11888.825672] LNet: Added LNI 192.168.202.153@tcp29 [8/256/0/180] [11888.829762] LNetError: 479958:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11888.833554] LNetError: Protocol error connecting to 192.168.202.153@tcp28 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11889.760192] LNet: 1 local NIs in recovery (showing 1): 192.168.202.153@tcp28 [11890.752155] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11890.762627] LNet: Added LNI 192.168.202.153@tcp30 [8/256/0/180] [11891.808989] LNetError: 479957:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11891.812431] LNetError: 479957:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [11891.814517] LNetError: Protocol error connecting to 192.168.202.153@tcp28 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11891.818634] LNetError: Skipped 1 previous similar message [11892.652363] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11892.663889] LNet: Added LNI 192.168.202.153@tcp31 [8/256/0/180] [11894.619803] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11894.632433] LNet: Added LNI 192.168.202.153@tcp32 [8/256/0/180] [11895.907505] LNetError: 479957:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11895.912058] LNetError: 479957:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [11895.915588] LNetError: Protocol error connecting to 192.168.202.153@tcp29 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11895.919609] LNetError: Skipped 3 previous similar messages [11896.626297] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11896.637067] LNet: Added LNI 192.168.202.153@tcp33 [8/256/0/180] [11898.682052] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11898.692706] LNet: Added LNI 192.168.202.153@tcp34 [8/256/0/180] [11901.012684] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11901.025854] LNet: Added LNI 192.168.202.153@tcp35 [8/256/0/180] [11903.202340] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11903.215489] LNet: Added LNI 192.168.202.153@tcp36 [8/256/0/180] [11904.097822] LNetError: 479960:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11904.101316] LNetError: 479960:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 6 previous similar messages [11904.104120] LNetError: Protocol error connecting to 192.168.202.153@tcp32 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11904.107593] LNetError: Skipped 7 previous similar messages [11905.478773] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11905.491757] LNet: Added LNI 192.168.202.153@tcp37 [8/256/0/180] [11907.717283] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11907.729886] LNet: Added LNI 192.168.202.153@tcp38 [8/256/0/180] [11909.996697] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11910.007345] LNet: Added LNI 192.168.202.153@tcp39 [8/256/0/180] [11912.355757] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11912.367603] LNet: Added LNI 192.168.202.153@tcp40 [8/256/0/180] [11914.825428] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11914.836061] LNet: Added LNI 192.168.202.153@tcp41 [8/256/0/180] [11917.276711] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11917.289462] LNet: Added LNI 192.168.202.153@tcp42 [8/256/0/180] [11919.911411] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11919.923719] LNet: Added LNI 192.168.202.153@tcp43 [8/256/0/180] [11920.481450] LNetError: 479960:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11920.485530] LNetError: 479960:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 31 previous similar messages [11920.488226] LNetError: Protocol error connecting to 192.168.202.153@tcp41 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11920.491285] LNetError: Skipped 30 previous similar messages [11922.462964] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11922.474716] LNet: Added LNI 192.168.202.153@tcp44 [8/256/0/180] [11924.991336] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11925.004136] LNet: Added LNI 192.168.202.153@tcp45 [8/256/0/180] [11927.627446] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11927.641606] LNet: Added LNI 192.168.202.153@tcp46 [8/256/0/180] [11930.554516] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11930.569222] LNet: Added LNI 192.168.202.153@tcp47 [8/256/0/180] [11933.651166] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11933.664372] LNet: Added LNI 192.168.202.153@tcp48 [8/256/0/180] [11936.761959] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11936.774441] LNet: Added LNI 192.168.202.153@tcp49 [8/256/0/180] [11939.974406] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11939.988190] LNet: Added LNI 192.168.202.153@tcp50 [8/256/0/180] [11948.515910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11948.516566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.518928] LustreError: Skipped 1 previous similar message [11948.525175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11948.528666] Lustre: Skipped 1 previous similar message [11950.176201] LNet: 6 local NIs in recovery (showing 6): 192.168.202.153@tcp49, 192.168.202.153@tcp41, 192.168.202.153@tcp38, 192.168.202.153@tcp43, 192.168.202.153@tcp32, 192.168.202.153@tcp31 [11953.634595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11953.637185] Lustre: Skipped 1 previous similar message [11954.028079] LustreError: 504881:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11954.065616] Lustre: server umount lustre-OST0000 complete [11954.272941] LNetError: 504605:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.153 [11954.276916] LNetError: 504605:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 50 previous similar messages [11954.280408] LNetError: Protocol error connecting to 192.168.202.153@tcp43 at host 192.168.202.153:988: is it running a compatible version of Lustre? [11954.284484] LNetError: Skipped 50 previous similar messages [11958.240790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11958.243893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.247927] Lustre: Skipped 1 previous similar message [11958.249363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11961.475162] LustreError: 505083:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11961.476645] LustreError: 505083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11961.509551] Lustre: server umount lustre-MDT0000 complete [11962.786581] LustreError: 480690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353753 with bad export cookie 2655171764188545896 [11962.791383] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11962.792334] LustreError: 480690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11962.863564] LustreError: 505284:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11962.866599] LustreError: 505284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11962.923229] Lustre: server umount lustre-MDT0001 complete [11966.612609] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [11967.718595] Key type lgssc unregistered [11967.901590] LNet: 505823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11967.905884] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11967.908117] LNet: Removed LNI 192.168.202.153@tcp [11968.928214] LNet: Removed LNI 192.168.202.153@tcp10 [11968.929808] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11968.933267] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11968.936419] LNet: Removed LNI 192.168.202.153@tcp11 [11969.952147] LNet: Removed LNI 192.168.202.153@tcp12 [11969.954140] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11969.957328] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11969.961834] LNet: Removed LNI 192.168.202.153@tcp13 [11970.976124] LNet: Removed LNI 192.168.202.153@tcp14 [11972.000177] LNet: Removed LNI 192.168.202.153@tcp15 [11972.001534] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11972.003882] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11972.007355] LNet: Removed LNI 192.168.202.153@tcp16 [11973.024171] LNet: Removed LNI 192.168.202.153@tcp17 [11974.048146] LNet: Removed LNI 192.168.202.153@tcp18 [11975.072218] LNet: Removed LNI 192.168.202.153@tcp19 [11976.096164] LNet: Removed LNI 192.168.202.153@tcp20 [11976.098044] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11976.100840] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11976.104832] LNet: Removed LNI 192.168.202.153@tcp21 [11977.120138] LNet: Removed LNI 192.168.202.153@tcp22 [11978.144131] LNet: Removed LNI 192.168.202.153@tcp23 [11979.168266] LNet: Removed LNI 192.168.202.153@tcp24 [11980.192138] LNet: Removed LNI 192.168.202.153@tcp25 [11981.216131] LNet: Removed LNI 192.168.202.153@tcp26 [11982.240164] LNet: Removed LNI 192.168.202.153@tcp27 [11983.264179] LNet: Removed LNI 192.168.202.153@tcp28 [11984.288131] LNet: Removed LNI 192.168.202.153@tcp29 [11984.289714] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11984.292689] LNetError: 505823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11984.295453] LNet: Removed LNI 192.168.202.153@tcp30 [11984.296512] LNet: Removed LNI 192.168.202.153@tcp31 [11984.297588] LNet: Removed LNI 192.168.202.153@tcp32 [11985.312177] LNet: Removed LNI 192.168.202.153@tcp33 [11986.336141] LNet: Removed LNI 192.168.202.153@tcp34 [11987.360215] LNet: Removed LNI 192.168.202.153@tcp35 [11988.384113] LNet: Removed LNI 192.168.202.153@tcp36 [11989.408115] LNet: Removed LNI 192.168.202.153@tcp37 [11989.409285] LNet: Removed LNI 192.168.202.153@tcp38 [11990.432153] LNet: Removed LNI 192.168.202.153@tcp39 [11991.456132] LNet: Removed LNI 192.168.202.153@tcp40 [11991.457862] LNet: Removed LNI 192.168.202.153@tcp41 [11992.480166] LNet: Removed LNI 192.168.202.153@tcp42 [11992.481589] LNet: Removed LNI 192.168.202.153@tcp43 [11993.504138] LNet: Removed LNI 192.168.202.153@tcp44 [11994.528159] LNet: Removed LNI 192.168.202.153@tcp45 [11995.552204] LNet: Removed LNI 192.168.202.153@tcp46 [11996.576139] LNet: Removed LNI 192.168.202.153@tcp47 [11997.600154] LNet: Removed LNI 192.168.202.153@tcp48 [11998.624188] LNet: Removed LNI 192.168.202.153@tcp49 [11998.631418] LNet: Removed LNI 192.168.202.153@tcp50 [11998.923625] Key type .llcrypt unregistered [11998.924709] Key type ._llcrypt unregistered [12006.917104] Key type ._llcrypt registered [12006.918137] Key type .llcrypt registered [12006.958754] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12007.281030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12007.323903] alg: No test for adler32 (adler32-zlib) [12008.203534] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [12008.305656] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [12009.896504] Key type lgssc registered [12010.323362] Lustre: Echo OBD driver; http://www.lustre.org/ [12013.243512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12013.250762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.429410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12015.875030] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12018.984873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12019.116045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12020.477771] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12022.379327] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12023.627785] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12025.891260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12026.006976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12027.863966] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12029.627843] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12043.550463] loop0: detected capacity change from 0 to 204800000 [12043.610189] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12043.615195] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12043.619959] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12043.647986] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12045.762577] loop0: detected capacity change from 0 to 204800000 [12045.828960] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12045.833356] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12045.838258] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12045.890233] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12047.634723] loop0: detected capacity change from 0 to 204800000 [12049.546260] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12049.568444] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12049.612842] Lustre: fs15246-OST0000: new disk, initializing [12049.614853] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12049.634183] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12051.562432] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12055.528191] loop1: detected capacity change from 0 to 204800000 [12057.496098] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12057.521599] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12057.555125] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12057.580967] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12057.620189] Lustre: fs15246-MDT0000: new disk, initializing [12057.676620] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12057.686951] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12057.729231] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12058.784787] Lustre: 511445:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000425da976 x1849010472123264/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763353865 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12059.060801] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12067.872265] LustreError: 511445:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [12067.876614] LustreError: 511445:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [12067.880601] LustreError: 511445:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [12071.910706] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12071.914955] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12072.955986] LustreError: 512564:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [12072.981085] Lustre: server umount fs15246-MDT0000 complete [12076.036189] LustreError: 513068:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [12076.038446] LustreError: 513068:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12076.052861] Lustre: server umount fs15246-OST0000 complete [12080.610156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12080.610841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12080.613447] LustreError: Skipped 1 previous similar message [12080.618242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12080.621604] Lustre: Skipped 1 previous similar message [12082.147412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12082.150545] Lustre: Skipped 1 previous similar message [12085.485730] LustreError: 513569:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12085.487757] LustreError: 513569:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12085.526825] Lustre: server umount lustre-OST0000 complete [12087.265712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12087.268804] Lustre: Skipped 1 previous similar message [12087.270523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12087.272725] Lustre: Skipped 1 previous similar message [12092.384931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12092.386514] Lustre: Skipped 1 previous similar message [12101.088105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12101.123288] LustreError: 513773:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12101.125597] LustreError: 513773:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12101.150217] Lustre: server umount lustre-MDT0000 complete [12102.400214] LustreError: 513775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353892 with bad export cookie 12813242632022007928 [12102.403344] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12102.403377] LustreError: 513775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12102.527052] Lustre: server umount lustre-MDT0001 complete [12106.119244] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [12107.156674] Key type lgssc unregistered [12107.281342] LNet: 514510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12107.284534] LNetError: 514510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12107.293350] LNet: Removed LNI 192.168.202.153@tcp [12107.590238] Key type .llcrypt unregistered [12107.591168] Key type ._llcrypt unregistered [12114.949855] Key type ._llcrypt registered [12114.951159] Key type .llcrypt registered [12114.989817] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12115.404432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12115.416509] alg: No test for adler32 (adler32-zlib) [12116.281372] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [12116.368534] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [12117.952134] Key type lgssc registered [12118.327534] Lustre: Echo OBD driver; http://www.lustre.org/ [12120.701154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12122.331143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12124.000130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12125.614070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12130.829065] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [12133.860652] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12137.341411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12139.519888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12141.519991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12143.578037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12145.833859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12145.855540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12145.860523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12146.940977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12146.953064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12146.987285] Lustre: lustre-MDT0000: new disk, initializing [12147.012563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12147.020533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12148.281689] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12152.428091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12152.450967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12152.482814] Lustre: 518914:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12152.507985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12152.510321] Lustre: Skipped 1 previous similar message [12152.548504] Lustre: lustre-MDT0001: new disk, initializing [12152.579451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12152.591607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12152.595220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12153.935708] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12157.150123] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12158.311357] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12160.602306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12160.625731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12160.716139] Lustre: lustre-OST0000: new disk, initializing [12160.718117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12160.748502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12162.617680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12162.621871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12162.641951] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [12162.659672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12165.687625] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12167.038740] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12167.108092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12168.419987] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12168.492533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12171.118119] LustreError: 521850:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12171.153639] Lustre: server umount lustre-OST0000 complete [12172.770290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12172.771127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12172.773228] Lustre: Skipped 1 previous similar message [12177.890767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12177.892688] Lustre: Skipped 2 previous similar messages [12181.989060] LustreError: 522052:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12181.991721] LustreError: 522052:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12182.021481] Lustre: server umount lustre-MDT0000 complete [12183.344454] LustreError: 518907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353973 with bad export cookie 2153785858392727297 [12183.348418] LustreError: 518907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12183.349693] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12192.741758] LustreError: 522254:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12192.743946] LustreError: 522254:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12192.803039] Lustre: server umount lustre-MDT0001 complete [12201.359717] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [12204.605165] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [12207.918993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12210.099710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12212.284866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12214.419095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12215.004674] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11962 sec ========= 23:33:23 (1763354003) [12215.580646] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:33:24 (1763354004) === [12216.777709] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:33:25 (1763354005) === [12229.431377] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [12230.452911] Key type lgssc unregistered [12230.574420] LNet: 527104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12230.577418] LNetError: 527104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12230.587305] LNet: Removed LNI 192.168.202.153@tcp [12230.887191] Key type .llcrypt unregistered [12230.888727] Key type ._llcrypt unregistered